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Multiple Object Tracking System 

Field of Invention 

The present invention relates to systems for tracking the movement of multiple 
objects within a predefined area. 

5 Background of the Invention 

As camera, micro-electronic and computer system technology continues to 
advance at a great pace there has been an increasing supply of machine vision systems intended 
to replace well defined, repetitive vision / recognition tasks which had previously been 
performed by humans. Early systems were designed to recognize parts moving along assembly 

10 lines to aid the manufacturing process. More recently, many inventions have been put forth to 
recognize humans and there movement. The variability of humans and their clothing as well as 
the complexity of the backgrounds within which they endeavor has presented a significant 
challenge to state-of-the-art technology. Considerable attention has been paid to various 
techniques for discerning the human shape from its background using edge detection 

15 techniques that look to remove stationary, i.e. background, information. There are two major 
factors that effect the success of these various techniques. The first is image resolution, which 
drives the amount of information and hence detail that is available to the accompanying 
computer system in order to differentiate foreground from background. Ideally the greater the 
resolution the better. However as resolution increasing so does the cost of the camera as well 

20 as the accompanying computer system. Even more importantly, as resolution increases time to 
process increases significantly. And as time to process increases the ability for these systems to 
perform real-time operations is impeded. 

The following are seven examples of machine vision systems designed in some 
way or another to recognize human or object motion from within a predefined area. 

25 In November of 1994 U.S. Patent 5,363,297 entitled Automated Camera-Based 

Tracking System for Sports Contests issued to Larson et al. This system employed multiple 
cameras to continuously monitor the area of an ongoing sporting event. Each camera would 
feed its information to an accompanying computer system for analysis that consisted of 
extracting the players from the stationary background and thereby tracking their silhouettes. 
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The inventor had anticipated problems when the individual players would collide or otherwise 
engage each other and hence merging their individual silhouettes. A need was also recognized 
to initialize the system by first identifying each player as they appeared within the system's 
field of view. Larson et al. specified two solutions for these problems. First, they proposed 
5 attaching monitors to the tracking system that would be operated by humans who would 

perform the initial recognition as well as all subsequent re-identifications when the system lost 
track of a player due to a merging of silhouettes. Secondly, they proposed attaching electronic 
tracking devices and employing triangulation of received signals in order to identify and track 
individual players. There are at least four major problems with the Larson patent. First, the 
10 amount of digital processing required to perform the player extraction in real time greatly 

exceeds the cost-effective computer technology of today, let alone 1994. Secondly, in order to 
perform this extraction, a greater amount of detail would be necessary which would thereby 
increase the cost of implementation by requiring more cameras and related computer systems. 
And of course the additional detail would only tend to further slow down the responsiveness of 
15 the system. Thirdly, the requirement of one or more operators to initially recognize and then 
re-identify players is extremely limiting and costly. This requirement essentially made the 
patent economically impractical for monitoring non-professional youth sporting events, where 
the system cost including the ongoing cost of the human operator would greatly exceed smaller 
revenue streams. It should be noted that this operator would more than likely be a parent of 
20 one of the youths who would probably be unfamiliar with all of the players and who would 

more than likely find it stressful to make so many decisions in real time. This approach would 
also require training and retraining operators, which would also be prohibitive. Fourthly, the 
type of electronics necessary to track players in real time would have to operate at higher 
frequencies that would also mean that it would be more expensive, providing a further 
25 economic drawback. The fifth major problem is that the system could have a difficult time 

determining the orientation of the player. For example, while Larson's invention could detect 
the direction of an individual's motion, it could not determine if the person was facing 
forwards or backwards, let alone if the person's head was turning in another direction. 

In December of 1995, U.S. patent number 5,473,369 entitled Object Tracking 
30 Apparatus issued to Keiko Abe. This system was concerned with the actual image processing 
techniques used to follow an object from frame to frame. The inventor described prior art that 
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compared the images block by block from one frame to the next, where a block is assumed to 
be one or more pixels of the image. It was pointed out that such systems depended upon time 
consuming and error prone statistical calculations that were especially susceptible to 
misinterpretation when the object changed size within the field of view or disappeared 

5 altogether. Abe proposed taking the same video frames but first separating them into luminance 
and color histograms that are then to be compared frame by frame. By comparing the 
histograms rather than blocks Abe argued that the system would be more accurate and efficient 
that the block matching systems. However, there are at least five major problems with Abe's 
patent. First, the effectiveness and reliability of this technique is highly dependent upon the 

10 lighting conditions initially and over time within the field of view being tracked. For instance, 
if the initial frame was taken under a well-lit condition, the luminance histograms of the object 
may be ideal. However, when the lighting conditions are poor to begin with, or worse yet 
change from frame to frame as might happen with sudden burst of ambient light, the luminance 
histograms will be subject to considerable error. Secondly, relying upon color histograms is 

15 equally uncertain due partly to the susceptibility of color detection to lighting conditions, which 
again may vary from frame to frame, and in part to the potential for object and background 
blurring when color schemes overlap. A third problem is that Abe's system does not lend itself 
to tracking multiple objects that may have similar or identical luminance / color information 
and which may overlap from frame to frame. A forth problem is discussed in Abe's 

20 specification that indicates a requirement of a human operator to initialize the system by 

selecting the portion of the video image which contains the object to be tracked, his so-called 
region designating frame. This requirement would be even more restrictive when consideration 
is given to multiple object tracking where the objects may go in and out of the field of view of 
temporarily overlap each other. And finally, a fifth problem is alluded to in the specification 

25 where it is expressed as an opportunity of the system to automatically control the pan, tilt and 
zoom of a camera. In so doing, Abe states that the system is capable of "coping with any 
change in the in the size of the object and which can photograph the target object always in a 
desirable size, thereby attaining a substantial improvement in terms of the facility with which 
the apparatus can be used." Hence, it is recognized that this method / apparatus is still very 

30 resolution dependent similar to the block methods it is attempting to improve upon. 



BNSDOCID: <WO 0031 560 A2 I > 



WO 00/31560 



PCT/US99/27281 



-4- 

In April of 1997, U.S. Patent Number 5,617,335 entitled System for and 
Method of Recognizing and Tracking Target Mark issued to Hashima et al. This invention is 
attempting to address the problem of determining the three-dimensional coordinates of an 
object with respect to a tracking camera and processing mechanism, e.g. a robotic arm from a 

5 single two-dimensional image. He expresses these coordinates as the attitude and position of a 
target mark that has been placed upon the object to be tracked. In Hashima's review of the 
prior art he lists several existing methods many of which require too many calculations and or 
have problems with multiple objects and background image noise. He discloses a technique for 
marking the object to be tracked with a white triangle inside of a black circle. Once the special 

10 markings are captured, they are quickly converted into projected histograms in the X and Y 
directions of the image of the triangle mark after which the centers of gravity as well as the 
maximum histogram values in the X and Y directions are also determined. All of this 
information is then collectively used for "determining which of classified and preset attitude 
patterns the attitude of the triangle of said target mark belongs to based upon the position of the 

15 centers of gravity, the maximum histogram values, the X and Y-axis values, and the known 

geometrical data of said target mark". Even taking Hashima's assertion of increased efficiency 
and accuracy, his technique has at least three major limitations. First, the object to be tracked 
must be marked in a highly accurate fashion and this mark must be visible to the tracking 
camera at all times. No provision has been disclosed at to how the object can be tracked if the 

20 markings are temporarily blocked from the view of the tracking camera. Secondly, by 

attempting to determine three dimension information from a single two dimensional image, 
Hashima is focusing his solution on situations were additional perspective cameras may not be 
available. Given such additional cameras, there exist even more efficient and accurate methods 
for determining the third dimension. Thirdly, this invention teaches of a system which 

25 functions well "even when an image either contains many objects or has a lot of noises" . 

However, if every one of these multiple objects needed to be tracked within the same image, 
Hashima's invention would not be at optimal performance since at any given time the preferred 
orientation of camera to object cannot be simultaneously maintained for multiple objects 
scattered in three dimensions. 

30 In March of 1998, U.S. Patent Number 5,731,785 entitled System and Method 

for Locating Objects Including an Inhibiting Feature issued to Lemelson et al. This invention 
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teaches the tracking of objects by "an electronic code generating system or device carried by 
the object in a portable housing". This "system or device" is specified to receive locating 
signals such as from the GPS constellation or a ground based triangulation setup. It then uses 
these signals to determine it's own location. Lemelson anticipates that at some point in time 
5 operators of a remote tracking system may be interested in the exact whereabouts of one 
individual object from amongst the multiplicity of objects that are housing such tracking 
devices. In order to determine the objects location, the tracking system will first transmit a 
unique "inquiry signal" coded for one particular device in one particular object. All of the 
individual tracking devices will then receive this signal but only the one device whose 

10 identifying code matches the "inquiry signal" will respond. This response is in the form of a 
transmission that includes the tracking devices currently determined location. The tracking 
system then receives this signal and displays on a computer system monitor related information 
about the identified / located object. Lemelson et al.'s invention is primarily applicable to the 
tracking of many objects over a very wide area, so wide that these objects are out of range of 

15 any reasonably sized camera tracking system. As an apparatus and method for tracking objects 
that are within a range suitable for a camera network, this invention has at least three major 
problems. First, it requires that each object have the capability to constantly monitor and track 
its own location. Such a requirement involves the use of a computing device which must be set 
up to receive GPS or other tracking signals and also transmit locating signals. Such a device 

20 will typically take up more space than a marking or indicia that can be placed on an object and 
then tracked with a camera. Furthermore, this device will require power. Secondly, Lemelson 
et al.'s invention assumes that the remote tracking station is only interested in one or a smaller 
fraction of all potential objects at a given time. However, there are many situations when it is 
desirous to follow the exact and continuous movements of all tracked objects as they move 

25 about within a predefined area. Whereas it is conceivable that this invention could constantly 
transmit "inquiry signals" for all objects and constantly receive locating signal responses, it is 
anticipated that this amount of information would unacceptably limit the movement resolution 
of the system. Thirdly, such an electronic based system has no understanding of an object's 
orientation with respect to it's direction of movement. Hence, while it is possible to determine 

30 the direction a car or person being tracked is moving, it is not shown how the system could 
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determine if that same car or person was facing or turned away from it's current direction of 
travel. 

In June of 1998, U.S. Patent No. 5,768,151 entitled System for Determining 
the Trajectory of an Object in a Sports Simulator issued to Lowery et al. This invention teaches 
the use of stereoscopic cameras trained upon a limited field of view to follow the trajectory of 
an anticipated object. As the object transverses the field of view the cameras capture images at 
a suitably slower rate such that the object creates a blur as it moves. This blurred path is then 
analyzed and converted into the object's trajectory vectors within the field of view. Another 
key means of Lowery et al.'s apparatus is its ability to determine when it should begin 
tracking. As such a sound detecting device is specified to sense the presence of the object 
within the field of view after which the image capture system is then immediately activated. 
There are at least four major limitations with Lowery et al.'s invention that would hinder its 
broader applicability. First, this invention expects a very narrow range of motion of the object 
and as such has a significantly restricted field of view. If the concept were to be applied to a 
larger area then multiple perspective cameras would need to be employed. The system would 
then also need to determine which cameras should be activated once the object is detected to be 
traveling within the tracking region. However, without actually first determining where the 
object is located, say for instance by attempting to triangulate the sound emitted by the object, 
the system would have no idea which cameras to activate. Hence all cameras would need to 
capture images creating a very large set of data that would need to be parsed by the tracking 
computer in order to determine the location of the object. The second limitation is that this 
system does not attempt to uniquely identify each object that it senses. Hence, while it is 
capable of determining a trajectory vector for individual objects, it does not anticipate a need 
for, nor disclose a method of determining the unique identity of each object as it passes by. 
The third limitation deals with the ability to track multiple objects simultaneously within this 
same field of view. Since this invention anticipates only one object at a time, it is merely 
determining trajectory vectors, and not the object's identity. Hence if two or more objects are 
traveling throughout the tracking region and they collide in such a way as to effect each other's 
path of travel, then the system will be left to determine which object continued on which path 
after the merge event. The forth limitation deals with the systems inability to pick up the object 



BNSDOCID: <WO 0031 560A2_I_> 



WO 00/31560 



PCT/US99/27281 



from its background when there is insufficient color and or luminescence difference between 
the two. 

All of the above listed prior art in one way or another was attempting to track 
the movement of at least one object within a predefined area. When taken in combination, their 
5 limitations that must be overcome in total are as follows: 

1- If the tracking system attempts to differentiate between the object and 
its background purely on the basis of pixel by pixel comparison as does Larson et al., then the 
video image must have higher resolution to be accurate and the resulting computer processing 
time prohibits real time operation. 

10 2- If the tracking system attempts to reduce processing time by performing 

averaging techniques based upon separated color and luminescence information as does Abe, 
then accuracy is compromised especially as colors merge between multiple objects and their 
background or lighting conditions fluctuate substantially between image frames. Such reduction 
techniques are further hampered as object size diminishes which essentially reduces the amount 

15 of object versus background information thereby increasing "noise". The only solution is to 
zoom in on the object being tracking to keep the proper ratio of object to background 
information. This then implies that each object being tracked must have its own camera thereby 
greatly reducing the effectiveness of these techniques for tracking either more objects and / or 
greater fields of view. 

20 3- If the tracking system such as Lowery et al.'s employs two perspective 

cameras and an image blurring technique to capture three-dimensional trajectory information it 
reduces image processing requirements but looses important video image detail. 

4- If the tracking system such as Hashima et al.'s employs detailed indicia 
placed upon the object to be tracked, this can be effective to reduce the amount of image 
25 processing. However, Hashima faces significant issues when trying to determine three- 
dimensional information from a single two-dimensional image, which is one of his mandatory 
requirements. His resultant techniques preclude the tracking of multiple fast moving objects 
over wider fields of view where the object's indicia may at times be blocked from view or at 
least be at significantly changing perspectives from the tracking camera. 
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5- All of the video / camera only based techniques such as Larson, Abe, 
Hashima and Lowery are prone to error if they were to track multiple objects whose paths 
would intersect and / or collide. Only Larson specifically anticipates this type of multiple 
object tracking and suggests the use of human operators to resolve object overlap. Such 
operators are cost prohibitive and also limited in their capacity to keep up with multiple fast 
moving objects in real time. While, as Larson suggests, it is possible to use passive electronics 
to help identify objects once the system determines their identities have been lost, these devices 
will have their own resolution / speed of response restrictions which are cost sensitive. 

6- Furthermore, both Larson and Abe's video / camera solutions anticipate 
the requirement of a human operator to initialize the system. Larson would require the operator 
to identify each object for the system. These objects would then be automatically tracked until 
they merged in some way with another object at which time the operator would be needed to 
re-initialize the tracking system. Abe would require the operator to crop the initial image down 
to a "region designating frame" which essentially reduces the processing requirements to at 
least find if not also track the object. The intervention of any operator is both cost prohibitive 
and real-time-response limiting. 

7- Lowery' s video / camera solution anticipates automatic tracking 
activation based upon the sound detected presence of an object within the field of view. This 
technique is inherently limited to objects that make distinguishing sounds. It is also unable to 
track multiple objects that might be making similar noises simultaneously within the given field 
of view. 

8- If the tracking system attempts to eliminate image processing 
requirements by employing active electronic tracking devices such as Lemelson et ah, then the 
objects are required to house powered devices capable of receiving and processing both 
locating and inquiry signals. Such devices limit the range and type of objects that can be 
tracked based upon the practicality and cost of embedding computing devices. In systems such 
as Lemelson's that employ electronics alone, the wealth of information available from image 
data is lost. Furthermore, such systems may be able to track location but cannot track object 
orientation, e.g. is the object traveling forwards or backwards. 
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9- With the exception of Hashima's indicia technique, all of these solutions 
are still not capturing object orientation information. Such information can be extremely 
important to anticipate future object movement. 

10- All of the video / camera based solutions will have difficulty picking up 
5 fast moving objects whose color and or luminescence information is sufficiently close to that of 

other tracked objects or the image background no matter what technique is employed. All non- 
video based solutions will give up valuable image information. 

Summary of the Invention 

While the present invention will be specified in reference to one particular 
10 example of multi-object tracking as will be described forthwith, this specification should not be 
construed as a limitation on the scope of the invention, but rather as an exemplification of 
preferred embodiments thereof. The inventors envision many related uses of the apparatus and 
methods herein disclosed only some of which will be mentioned in the conclusion to this 
applications specification. For purposes of teaching the novel aspects of this invention, the 
15 example of multi-object tracking is that of a sporting event such as hockey. The particular 
aspects of hockey which make it a difficult series of events to track and therefore a good 
example of the strengths of the present invention over the prior art are as follows: 

1- There are no other human based activities know to the present inventors 
where the humans as objects can travel at a faster speed or change directions and orientation 
20 more quickly than hockey. On skates a player's speed can approach twenty-five miles per 
hour, which is considerably faster than any activity involving walking or running that is still 
conducted on the ground without the aid of a vehicle of some sort. Tracking these faster 
movements, especially given the variability of the human form, challenges the real time 
performance aspects of the system. 

25 2- The speed of the object being contested by the players, i.e. the puck, 

can travel at rates of up to one hundred miles per hour and is also subject to sudden and quick 
changes of direction. This combination of speed and re-direction presents a difficult tracking 
problem and is unique in athletics. Tracking the puck is easier than tracking player when 
considering that the puck will not change shape yet the puck travels at roughly four times the 

30 speed, is on the magnitude of one hundred times as small and may travel in three dimensions . 
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S' The individual players are constantly entering and exiting the tracking 
field of view and as such must be efficiently and automatically identified by the tracking system 
in order for real time performance. 

4- While in the field of view, both the puck and the players are routinely 
5 either fully or partially hidden from view as they merge with the paths of other players. This 

creates a challenge to follow movements with often limited or no image data. 

5- The lighting conditions are difficult to work with since the ice surface 
will create a highly reflective background that could tend to saturate the CCD elements of the 
camera while the area itself may be subject to sudden bursts of light from either spectator's 

10 camera flashes or in-house lighting systems. This places limitations on luminescence based 
tracking techniques. 

6- The colors of the players on the same team are identical and may often 
match the markings on the ice surface and surrounding rink boards. This places limitations on 
color based tracking techniques. 

15 7- It is not unusual for a hockey game to be played while a certain level of 

fog exists within the arena. This challenges any camera-based system since it could greatly 
reduce visibility of the players and puck. 

8- Hockey is a filmed event and as such it presents the opportunity not 
only to track the movement of multiple objects but also to determine a center of interest which 

20 is constantly and abruptly changing. Once this center is determined, there is a further 

advantage to automatically direct the tilt, pan and zoom of a broadcaster's camera to follow the 
action from a perspective view in real time. Automatically directing a camera that is situated 
for a perspective view presents a difficult problem for a machine vision system since it is 
considerably harder to follow the objects in three dimensions in real time as a perspective view 

25 would require. 

9- Each individual player as well as the coaches in a game may at any time 
be instantly desirous of obtaining information regarding themselves, a group of players, their 
entire team and / or the other team. This information may furthermore pertain to the entire or 
some sub-set of the duration of the activities from start to present. Such requirements place a 
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demand on the tracking system to quickly and efficiently store information in a way that it may 
easily be recalled from many viewpoints. 

10- The enclosed metal and cement block arena precludes the use of GPS 
and presents difficulties for the use of passive electronic tracking devices due to the many 

5 potential reflections of in-house triangulation signals. The players themselves and the nature of 
the game and its potential for significant high impact collisions limit the desirability of placing 
active electronic devices within their equipment. Since these devices must carry a power source 
they will in practice take up enough space to present a potential hazard to the players. 
Furthermore, such devices would be extremely cost prohibitive at the local rink level where 
10 literally hundreds of children are playing games every week and would each need their own 
devices or to share devices. 

11- Player orientation and location of limbs with respect to body are very 
important information to track. A player may be traveling forward facing forward while they 
are turning their head to the side to view the development of play. The turn of the head 

15 becomes important information for coaching analysis since it is an indication of a player's on- 
ice awareness. Furthermore, a play may almost instantly rotate their orientation while still 
proceeding in the same direction such that they are now traveling backwards instead of 
forwards. This is also very critical information. A player's head may be tilted down towards 
the ice, which is not desirable for prolonged periods of time or if often repeated. All of this 

20 information is important to track but presents problems over and above simply tracking the 
player as a whole. 

12- Limiting the size of the area of tracking is desirable during practice 
sessions where individual drills may be conducted on a limited portion of the ice with a small 
number of players at a time. Under these conditions it would be desirable to easily restrict the 

25 system's tracking area within its field of view. 

13- The number and speed of player changes and collisions is so great that 
utilizing human intervention to identify and re-identify players would be significantly stressful 
and error prone, if not practically impossible, especially at the local rink level. 

Given the current state of the art in camera systems, non-visible energy sources 
30 and filters, digital image processing and automated camera controls it is possible to create an 
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entirely automated multi-object tracking system which operates within a predefined area and 
tracks the constant location, orientation and direction of movement of each and every object 
within the field of view. Such a system greatly increases the ability of the participants and 
observers to understand, analyze and enjoy the given activity. 

5 Accordingly, the objects and advantages of the present invention are to provide 

a system for tracking multiple objects within a predefined area with the following capabilities: 

1- to provide a system for tracking multiple, fast moving, variably shaped 
objects such as humans in real time without the aid of operator intervention to either initially 
identify or conditionally re-identify the objects during the tracking period; 

10 2- to reduce the time required to process image data by first separately 

tracking a special indicia which has been attached to the object after which the indicia's 
location may then be used along with the vectors of direction, acceleration and velocity to 
efficiently extract the object from its background; 

3- to provide a system which is not dependent upon color distinctions 
15 within the objects; 

4- to provide a system which does not rely upon either passive or active 
electronic tracking devices; 

5- to provide a system which determines orientation information about the 
objects as a whole and potentially of their individual parts; 

20 6- to provide a system which can work under various lighting and air 

moisture conditions which may not be optimal for recognition techniques based upon visible 
light; 

7- to provide a system where the field of tracking is easily constricted 
within the systems field of view; 

25 8- to provide a system which can create and maintain in real time a 

movement database accessible by either time and or object(s) identity; 
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9- to provide a system which can output either all or some of the captured 
data either immediately within the ice arena of after the event to one or more remote computers 
over a communication network such as the Internet; and 

10- to provide a system such that the information output to remote computer 
5 systems over remote communications such as the Internet may be used for controlled event 

replay as well as critical analysis. 

Further objects and advantages are to provide a cost efficient system to build, 
install and maintain with a minimum of moving parts that is capable of operating under a range 
of temperature conditions. Still further objects and advantages of the present invention will 
10 become apparent from a consideration of the drawings and ensuing description. 

Description of the Drawings 

Fig. 1 is a top view drawing of the preferred embodiment of the present 
invention depicting an array of overhead X-Y tracking cameras which when taken together 
form a field of view encompassing the skating and bench area within an ice hockey arena. Also 
15 depicted are perspective Z tracking camera sets behind each goal, automatic pan, tilt and zoom 
perspective filming cameras as well as a single representative player and puck. 

Fig, 2a is a set of three perspective drawings depicting a typical player's jersey, 
pads with tracking patches in place and then a combination of the jersey over the pad with 
patches. 

20 Fig. 2b is a set of two perspective drawings depicting a hockey puck as well as 

a typical player's hockey stick, where each has been augmented to include tracking ink on at 
least some portion of their outer surfaces. 

Fig. 2c is a set of two perspective drawings depicting a typical hockey players 
helmet which has been augmented to include tracking stickers on at least some top portion of 
25 its outer surface. 

Fig. 3a is a perspective drawing of a typical hockey players pads, helmet, stick 
and puck being captured from an overhead X-Y filming camera and displayed on a viewing 
screen. 
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Fig. 3b is a perspective drawing similar to Fig. 3a except that now tracking ink 
has been added to the hockey stick and puck, tracking patches have been added to the pads and 
tracking stickers to the helmet. In addition, a tracking energy source as well as a frequency- 
matching filter has been added to the overhead X-Y filming camera making it a tracking 
5 camera. 

Fig. 4a is a perspective drawing similar to Fig. 3b except that now an 
additional un-filtered filming camera has been added to the overhead X-Y tracking camera so 
as to efficiently combine both sets of information. 

Fig. 4b is a top view depiction of a key element of the present invention's 
10 novel process for efficiently extracting the video image of the object being tracked by first 

locating an attached indicia and then working out away from the indicia comparing each pixel 
of the image to a pre-known background so as to effectively outline the tracked object. 

Fig. 4c is a top view of a portion of an ice arena showing a series of tracked 
and traced motions of a typical hockey player, stick and puck by the overhead X-Y filming and 
15 tracking cameras depicted in Fig. 4a. 

Fig. 5 is a perspective drawing of a mountable camera array frame with 
overhead X-Y filming and tracking cameras attached to the roof of a typical ice arena. 

Fig. 6 is a top view drawing of an ice arena where area restricting cones with 
tracking ink have been placed so as to indicate to the overhead X-Y tracking cameras that only 
20 a sub-portion of the entire field of view is to be tracked. Also depicted are typical players, one 
is within the tracking zone while many are outside of the zone. Further depicted is a portable 
device that may be used by on ice coaching staff to control the functions of and inquire upon 
the information generated by the present invention. 

Fig. 7 is a front view drawing of a hockey players bench which has been 
25 outfitted with a series of display devices with keypads that may be used by typical hockey 

players during the course of a hockey game to inquire upon the information generated by the 
tracking system. 

Fig. 8 is a block diagram depicting all of the tracking and computing elements 
of the proposed present invention. 
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Detailed Description of toe Invention 

Referring to Fig. 1 there is shown a top view drawing of the preferred 
embodiment of the Multiple Object Tracking System 100. System 100 comprises an array of 
overhead x-y camera assemblies 20c that individually track all object movement within a fixed 
5 area such as 20v. In total, the array of overhead assemblies 20c track all movements in ice 

playing surface 2, team boxes 2f and 2g, penalty box 2h as well as a portion of entrance-way 
2e. Assembly 20c further comprise filming camera 25, energy source 23, tracking camera 24 
onto which is attached energy filter 24f , all of which is housed in assembly casing 21 and has a 
view to the ice surface 2 below through assembly Plexiglas 21a. Energy source 23 emits 

10 selected energy 23a which radiates down onto surface 2 and off of the objects moving upon 
this surface such as player 10 and puck 3. It should be noted that selected energy 23a was 
specifically chosen to be of a frequency that not only is outside of the range of normal ambient 
light but also will not be effected by moisture conditions such as fog. A specially situated 
overhead assembly 20cl is shown as covering ice surface 2's entrance- way 2e. This assembly 

15 20cl is identical in construction to 20c and is important and will later be specifically referenced 
because all objects to be tracked must enter and exit through this camera's field of view. Also 
tracking movements on a selected portion of ice surface 2 are perspective z tracking camera 
sets 30 that are situated as one pair at both ends of the ice 2. And finally there are automatic 
filming cameras 40 which are constantly being directed to the center of play as represented by 

20 player 10 who is currently controlling puck 3. Automatic filming cameras 40 are in continuous 
communications with and are receiving their directions from local computer system for video 
processing and analysis 60. System 60 itself is also in continuous communications with array 
of overhead x-y tracking camera assemblies 20c and perspective z tracking camera sets 30. 
Local system 60 is in further in optional communication with remote computer system for 

25 reviewing captured events 70 which has attached viewing monitor 27 that displays scene 28. 
Also in optional communication with local system 60 are team output stations 80 which 
provide information on the object movements tracked from the beginning of the event to the 
current moment. 

Referring now to Fig. 2a there is depicted a typical player's jersey 5 and 
30 player's shoulder pads 6. Affixed to pads 6 are right shoulder team patch 7r and left shoulder 
player patch 71. Patch 7r comprises orientation mark 7rl, which is an arrowhead pointing 
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away from the head towards the arm and team indicia 7r2, which is a unique bar code. Patch 
71 comprises orientation mark 711, which is also an arrowhead pointing away from die head 
towards the arm and player indicia 712, which is a unique number. It should be noted that the 
indicia on patches 7r and 71 are created from a frequency selective reflective material, 
5 preferably an ink. Also referring to Fig. 2a there is depicted jersey 5 placed over pads 6. Note 
that jersey 5 is also shown to be cut-away for a full view of underlying player patch 71. 

Also depicted in Fig. 2a is reflected energy 7m which is shown radiating 
though transmissive jersey 5. Such techniques for seeing through cloth are not new and were 
demonstrated by Sony Corporation when they placed a shirt over a license plate and then took 

10 two pictures, one without a filter and the other with a visible light filter. Since cloth reflects 
light in the visible spectrum so intensely, the plate could not be seen by the unfiltered image. 
But with the infrared pass filter only the infrared light was captured making the numbers and 
letters on the license plate visible. It should be noted that the present invention teaches the use 
of special frequency selective reflective material that will further enhance the contrast, i.e. 

15 signal to noise ratio, of the desired image above and beyond the filtering techniques discussed 
by Sony. 

Referring now to Fig. 2b there is shown a typical hockey puck 3 where its top 
surface (and in practice all outer surfaces) has been coated with a reflective ink 3a similar to 
the frequency selective reflective material used in patches 7r and 71. In response to a particular 
20 frequency of energy as would be emitted for example by energy source 23, ink 3a causes 

reflected energy 3b. Also shown is a typical hockey stick 4 where its blade has been wrapped 
with a special reflective hockey tape 4a that has been produced to include the same special 
reflective ink. In response to a particular frequency of energy as would be emitted for example 
by energy source 23, reflective tape 4a causes reflected energy 4b. 

25 Referring now to Fig. 2c there is shown both a top and perspective view of a 

typical hockey players helmet 8 where a reflective sticker 9 has been applied to its top surface 
and has been produced to include the same special reflective ink. In response to a particular 
frequency of energy as would be emitted for example by energy source 23, sticker 9 causes 
reflected energy 9a. 
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Referring now to Fig. 3a there is shown a first embodiment of the overhead x-y 
tracking camera assembly 20a. In this embodiment, assembly 20a has been limited to tracking 
camera 24 (without energy filter 24f) which is enclosed within assembly casing 21 and has a 
view to the ice surface 2 below through assembly Plexiglas 21a. There is depicted below 
5 assembly 20a unmarked player 10, unmarked stick 4 and unmarked puck 3. Also show is cable 
26 which attaches assembly 20a to local computer system 60 (not depicted), to remote 
computer 70 (also not depicted) and therefore to viewing monitor 27 that displays scene 28. 

Referring now to Fig. 3b there is shown a second embodiment of the overhead 
x-y tracking camera assembly 20b. In this embodiment, assembly 20b has been augmented to 

10 include energy source 23 that emits selected energy 23a which radiates down onto the objects 
moving upon the ice surface such as player 10, puck 3 and stick 4. Note that player 10's pads 
6 have been augmented to include right shoulder team patch 7r and left shoulder player patch 
71. Also note that Puck 3 now includes reflective ink 3a and that stick 4 has been wrapped with 
a special reflective hockey tape 4a. Scene 28 now depicts a much smaller set of information to 

15 be analyzed and tracked. 

Referring now to Fig. 4a there is shown a third and the preferred embodiment 
of the overhead x-y tracking camera assembly 20c. In this embodiment, assembly 20c has been 
augmented to include filming camera 25 that captures unfiltered images of all movement on the 
ice surface below. Fig. 4b depicts a key element of the present invention's novel process for 

20 efficiently extracting the video image of the object being tracked by first locating an attached 

indicia and then working out away from the indicia comparing each pixel of the image to a pre- 
known background 2r. Once it is determined that background pixels have been reached, the 
image-tracing path lOp continues around the object until a closed path is completed. Around 
this closed path lOp a minimum bounding rectangle lOr is calculated in order to quickly 

25 extract the portion of the video image containing the tracked object. Referring also to Fig. 4c 
there is depicted the processed motion of player 10 upon ice surface 2 using the preferred 
assembly 20c. Player 10 is shown to transverse through four movements points lOal, 10a2, 
10a3 and 10a4 carrying puck 3 along the way towards taking a shot at goal 2b within 
calculated shot triangle 2c. The center of ice surface 2 is shown as point 2a from which first 

30 movement angle lOalo and movement radius lOalr have been calculated to represent player 
10's first movement to point lOal. Also calculated are movement angle 31o and movement 
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radius 31r that describe puck 3's initial movement to point lOal. About center point 2a there is 
shown a maximum bounding circle 10b within which an object's next location is calculated to 
be reside based upon the objects last known position (in this case point 2a) as well as its last 
calculated vectors of direction, acceleration and velocity. And finally there is depicted the 
5 maximum initialization search rectangle lOe which covers the outermost portion of the field of 
view as tracked by overhead assembly 20cl which is situated over the only entrance-way 2e to 
ice surface 2. 

Referring now to Fig, 5 there is shown overhead mounting frame 50 that is 
variably attachable to differing constructions of rink girders 2d. Further attached to frame 50 is 
10 the array of overhead x-y tracking camera assemblies such as 23c that emit selected energy 23a 
down upon the ice surface 2 below. Assemblies 23c are connected to local computer system 
for video processing / analysis 60. 

Referring now to Fig. 6 there is depicted ice surface 2 whose tracking area has 
been restricted by the placement of four typical cones 11 which have been coated with a 

15 reflective ink 11a. In response to a particular frequency of energy as would be emitted for 
example by energy source 23, ink 11a causes reflected energy lib. Shown within restricted 
region 2t is a single player 10 whose current motion is to be tracked while he remains within 
region 2t, Shown outside of region 2t are multiple players such as lOi whose movements are 
not being tracked. Also shown are shot clock 91 and portable tracking control device 90 that 

20 are both in optional communication with local computer system 60. 

Referring now to Fig. 7 there is shown a close up view of team output stations 
80 that are attached to team boxes 2f (shown) and 2g, as well as penalty box 2h. 

Referring now to Fig. 8 there is depicted a block diagram of the entire system. 
With respect to overhead x-y tracking camera assembly 30c there is shown additional optional 

25 rf modem 25m which may be used in place of wire 26 (not depicted) to link filming camera 25 
to local computer system 60. Also shown is additional optional RF modem 24m which may be 
used in place of wire 26 (not depicted) to link tracking camera 24 to local computer system 60. 
Likewise additions are also depicted for perspective z tracking camera sets 30. Local system 60 
has been broken down into its constituent block parts starting with the video capture for 

30 tracking camera(s) unit 61 which accepts input from all tracking camera(s) 24 mounted within 
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overhead assemblies 30c or perspective sets 30. Capture unit 61 then feeds its data stream to 
indicia / object tracking analysis unit 62. In time parallel operation there is shown video 
capture for filming camera(s) unit 63 which accepts input from all filming camera(s) 25 
mounted within overhead assemblies 30c or perspective sets 30. Capture unit 63 then feeds its 
5 data stream to object / scene extraction analysis unit 64. Unit 64 also receives simultaneous 
input from indicia / object tracking analysis unit 62. Both extraction analysis unit 64 and 
tracking analysis unit 62 simultaneously provide their data streams to composition of multiple 
overlapping images into individual full view frames unit 65. Unit 65 also receives conditional 
input from portable tracking control device 90 and provides its data stream to both center-of- 

10 play / choice of view analysis unit 66 and storage of tracking and video frames indexed by 

object and frame no. / time sequence unit 67. Choice of view analysis unit 66 in turn provides 
real time directives to automatic filming cameras 40 through conventional wire or optional RF 
modem 45m. Such directives are then input to computer directable event filming camera 45. 
Filming camera 45 then feeds its data stream back to storage and tracking unit 67 via 

15 conventional wire or optional RF modem. Unit 67 subsequently provides its data stream to 

quantification and analysis of object movement stored in cross-linked database unit 68. Unit 68 
optionally conducts two-way communications with portable tracking control device 90, team 
output stations 80 and remote computer system for reviewing captured events 70. The optional 
link with device 90 will be made via Rf connection (not depicted) while the link with remote 

20 system 70 will be made via remote communications devices 69 and 79. The information 

provided by local system 60 to remote system 70 will be provided to the end user via remote 
access to stored database of tracking and filming frames as well as object movement 
quantification and analysis unit 71. 

Operation 

25 Referring first to Fig.'s 1 and 5, normal operation of the preferred embodiment 

commences after the system 100 has been properly installed at an ice arena such as 2. One of 
the more critical aspects of installation is the attachment of the array of overhead x-y tracking 
camera assemblies 30c to the roof of arena 2. Mounting frame 50 has been designed to 
variably attach to rink girders 2d in such as way that the subsequently attached array of 

30 overhead assemblies 30c forms an overlapping field of view across of the ice surface 2 below. 
It is preferable that each assembly 30c maintains a perpendicular position with respect to 
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surface 2. It is anticipated that either fixed lenses with pre-selected depths of field or 
electronically controllable zoomable lenses will be used to properly establish the overlapping 
fields. The overlap is important to ensure that any object to be tracked can be followed from 
camera to camera as it moves throughout the predefined area. As a part of the initial 
5 installation and setup of system 100, a calibration procedure will be run to define the limits of 
each filming cameras* 25 and tracking cameras' 24 field of view. Once known, the system will 
restrict its searching to non-overlapping regions in order to minimize duplicate processing. All 
assemblies 30c will either communicate with local system 60 via cables such as 26 or optional 
RF modems such as 25m and 24m. 

10 Referring now to Fig.'s 2a, 2b and 2c, the preferred embodiment provides for 

various methods of marking the objects to be tracked with a specially chosen frequency 
selective reflective material such as an ink. This ink is then used to embed into puck 3 as 
reflective ink 3a, to produce reflective tape 4a, to embed into markings of patches 7r and 71, 
and to produce reflective stickers 9 for helmets 8. It is also used to create cones 11 with 

15 reflective ink 11a as shown in Fig. 6. It is critical to the novelty of the present invention to 

note that this ink has been chosen specifically to reflect energy 23a as output by energy source 
23. This reflection has been depicted as 3b, 4b, 7m, 9a and lib and will be received by 
tracking cameras 24 through energy filter 24f . By using specially matched energy filter 24f on 
each camera 24, the amount of information required to be processed in order to follow all 

20 object motion is minimized thus greatly increasing the speed of determining the object's 
position. 

To further illustrate the novelty of this point, Fig.'s 3a, 3b and 4a have been 
created to dramatize the resultant effects of the addition of this tracking ink. Fig. 3a shows a 
conventional camera system capturing ambient light and showing its corresponding field of 

25 view as scene 28 on monitor 27. Note that in this view there is considerably more information 
that must be processed including the object such as player 10 and the background such as ice 
surface 2. By adding reflective materials as previously discussed in the form of 3a, 4a, 7r, 71 
and 9, Fig. 3b now shows a much reduced amount of information in scene 28. Again, this is 
accomplished in real time without the requirements for additional computer processing by the 

30 application of energy filter 24f to tracking camera 24 which was specially chosen to pass a 

narrow band of frequency as emitted by energy source 23 as selected energy 23a. Energy 23a 
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radiates throughout the predefined area such as ice surface 2 and is reflected back as 3b, 4b, 
7m and 9a. Orientation marks 7rl and 711, team indicia 7r2, player indicia 712 as well as 
reflective sticker 9 have been produced to include special markings which will be easily 
discernable to indicia / object tracking analysis unit 62. The size of these markings will be 
5 matched to the target resolution of tracking cameras 24 in overhead assemblies 30c. Hence, the 
lower the resolution vs. field of view, the larger the marking will have to be to facilitate easy 
recognition. The preferred embodiment also includes unfiltered filming cameras 25 that will 
capture all ambient frequencies of light, as depicted in Fig. 4a. However, as will be discussed 
at a later point, due to information provided by the tracking analysis, only a small portion of 
10 this data must be gone through in order to extract all relevant object information. 

Prior to running system 100 it will also be necessary to install perspective z 
tracking camera sets 30. One set will be mounted at each end of ice surface 2 so as to be in 
constant view of the area immediately surrounding each goal 2b. These sets 30 are so placed in 
anticipation of the travel path of puck 3 which is expected to leave surface 2 and to travel in the 

15 Z direction when approaching goals 2b. Note that due to the overhead nature of assemblies 
30c, they will be unable to pick up any object movement in the Z dimensions which is 
acceptable and even desirable since it reduces the amount of processing that must be performed 
to track all objects. However, in the restricted area nearest to goals 2b, namely defined by 
triangular area 2c, it can be very beneficial to track the Z element of puck 3's path of travel. It 

20 is important to note that sets 30 and the information they collect are not required to allow the 
main aspect of what the present inventors feel is novel concerning this invention. Specifically, 
the practice of following movements of specially marked objects and to apply this information 
to normal video images to quickly extract object information. 

Also optional and yet novel to the present invention is the inclusion of 
25 automatic filming cameras 40 at discretionary locations within ice surface arena 2. Heretofore, 
camera operators who intuitively determined the center-of-play controlled the filming for 
broadcast of events such as hockey and self directed the pan, zoom and tilt of their filming 
cameras to capture the restricted scene. Now that system 100 will have all information 
concerning the movement of objects on the ice surface 2, it is planned that local computer 
30 system 60, via its center-of-play / choice of view analysis unit 66, will automatically direct the 
pan, zoom and tilt of the automatic filming cameras 40. 
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Once all overhead x-y tracking camera assemblies 30c, all perspective z 
tracking camera sets 30 and all automatic filming cameras 40 have been correctly installed and 
calibrated and all objects to be tracked have been appropriately augmented with marks such as 
3a, 4a, 7r, 71 and 9, system 100 is ready for operation. The ensuing discussion will describe 
5 how the movements of a single player 10, stick 4 and puck 3 will be tracked from frame to 

frame during the duration of the event across the entire predefined field of view in the X-Y and 
Z dimensions. Following this, the present inventors will teach how additional players 10 with 
their sticks 4 may also be simultaneously tracked and how problems due to object overlap, 
object change in size, object disappearance from view, sudden changes in ambient lighting and 
10 otherwise reductions in visibility for example because of fog will be handled. 

Referring now to Fig.'s 1, 2, 4a, 4b, and 8, the tracking of multiple objects by 
system 100 commences after the system has been turned on and the first player 10 proceeds 
down entrance-way 2e and crosses into the field of view of the array of overhead assemblies 
30c. Within entrance-way 2e player 10 will first appear in a single frame captured by video 
15 capture for tracking camera(s) unit 61. Unit 61 will in total capture 40 plus frames per second 
(40 + fps) from each tracking camera in assembly 20c and will feed each frame to indicia / 
object tracking analysis unit 62. 

Note that given the current state of technology one camera controller board 
such as would be required within unit 61 can control up to eight cameras. The actual number 

20 of frames per second that needs to be captured is dependent upon the desired movement 

resolution. For instance, the fastest object in our example is the puck 3 that will at times travel 
up to 100 mph. At 40 frames per second, 5280 feet per mile and 60 * 60 = 3,600 seconds per 
hour the maximum distance of travel for the puck 3 per second is: (100 miles/hour * 5280 
feet/mile) / (3,600 sec/hour) = 146.7 feet. At a rate of 40 frames per second, the maximum 

25 distance traveled between frames would be 3.7 feet. Note that a player such as 10 on skates 

will obtain maximum speeds of around 25 mph, or one quarter of the speed of the puck. At 40 
frames per second, the maximum distance traveled by a player 10 would be approximately 
0.925 feet. At 20 frames a second, the puck 3 will travel no more than 8 feet at maximum 
speed while the player 10 will be limited to approximately 2 feet in distance. Considering that 

30 both player 10 and puck 3 seldom travel at maximum speed, movement resolution will be 

considerably higher in practice. Further note that there is an anticipated area, namely around 
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goal 2b where puck 3 is most likely to reach top speeds. Both the overhead camera assemblies 
20c and the perspective camera sets 30 can be operated at an increased frame capture rate to 
increase movement resolution in these specific zones. In any case, the present invention has the 
ability to trade off cost of computing power versus movement resolution by trading off the 
5 frame rate and the number of tracking cameras 24 that will be attached to a single computer 
and video capture card. It should be recognized that the use of multiple computers to gather 
and analyze data is considered to be neither novel to, nor a limitation of, the present invention. 

As each frame from tracking camera 24 of overhead assembly 20cl is accepted 
by analysis unit 62, the grey scale of each pixel is compared against a threshold value where 

10 those pixels exceeding the threshold indicate the presence of any form of the special mark such 
as 3a, 4a, 7r, 71 and 9. Since every player 10 must both enter and leave the ice surface 2 from 
entrance-way 2e which is always in view by assembly 20cl, and since the maximum 
movement between frames can be calculated based upon the anticipated maximum speed and 
the frame capture rate, it is possible to calculate a minimum number of pixels that must be 

15 completely searched in order to first detect the presence of a new player such as 10. This 
minimum number of pixels is shown as lOe in Fig. 4c and consists of those pixels running 
parallel along the outermost edge of assembly 20cl's field of view inward several rows toward 
the main ice surface 2. The depth of this rectangular area will again depend upon the maximum 
distance a player 10 can be expected to travel between frame captures. Note that even in other 

20 sporting events such as basketball or football, there is usually a tunnel through which each 
team must travel to gain access to the main arena. If the systems field of view is properly 
extended to be continuous into any such tunnel, then this minimum search technique may be 
employed to first detect any new player to be tracked. As will be shown, once detected, the 
processing requirements of continuing to track are significantly reduced since the extent of a 

25 player's movement between frames is limited. 

Referring now additionally to Fig. 4c, once a marked object is detected, 
whether it be a patch 7r or 71 on pads 6, or the reflective sticker 9 on helmet 8, or the 
reflective ink 3a on puck 3, or the reflective hockey tape 4a wrapped around a stick blade 4, it 
is tracked individually. The first detected movement of each mark is expressed in polar 
30 coordinates relative to the center point 2a of ice surface 2. Hence each object's first movement 
is expressed as an angle off 0° and a distance from center point 2a along the stated angle, these 
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calculations are performed constantly by object tracking analysis unit 62. Once unit 62 has 
detected a given object, it will continue to look for that object within its field of view based 
upon the last known coordinates, the last know vectors of direction, acceleration and velocity 
and the calculated maximum distance of travel between frames. The last known coordinates 
5 combined with the maximum possible travel distance will work to define a maximum bounding 
circle, depicted as 10b in Fig. 4c, of possible movement which must be searched in order to 
find the objects next location. In order to cut the average search time within this bounding 
circle, unit 62 will first look in the last known direction of travel based upon prior movements 
out a distance equal to the last know rate of speed divided by the known frame rate. From this 

10 new most likely point within the bounding circle, unit 62 will continue to search by growing 
this point in all directions until the entire bounding circle has been analyzed. In the event that 
the marking is not found and it is known that the object has entirely passed through the 
minimum rectangle of pixels bordering the edge of the field-of-view covering the entrance-way 
2e, then the object will be searched for on the next frame. In this case the new maximum 

15 bounding circle's radius will be twice that of the previous search. 

If a marked object is detected and then is lost to system 100, tracking analysis 
unit 62 will first communicate with object / scene extraction analysis unit 64 to determine if the 
presence of additional ambient light based information will uncover the objects location. Before 
reviewing this technique, it should first be noted that once a marked object is detected, its type 

20 and location are passed to extraction unit 64 from tracking unit 63. extraction unit 64 then 
analyses the corresponding unfiltered video frame taken by the filming camera 25 that is 
housed in the same overhead assembly 20c whose tracking camera 24 is currently viewing the 
located object. Knowing the type of object, for example a patch 7r or reflective sticker 9, 
indicates to extraction unit 64 the maximum expected size of the actual object, for example 

25 pads 6 or helmet 8. Given this maximum size along with the current location of the reflective 
indicia, unit 64 will begin at the location of the attached indicia and then work its way out 
away from the indicia comparing each pixel of the image to a pre-known background 2r, as 
depicted in Fig. 4b. Once it is determined that background pixels have been reached, the 
image-tracing path lOp continues around the object until a closed path is completed. Around 

30 this closed path lOp a minimum bounding rectangle lOr is calculated in order to quickly 
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extract the portion of the video image containing the tracked object. By knowing the expected 
maximum size of the object associated with the type of indicia found object (e.g. pad 6 or 
helmet 8 or player 10), this process can be restricted to a maximum bounding circle in order to 
find the edge of the object. Note that this object defined by closed path lOp may and often will 

5 contain two or more indicia such as patches 7r and 71 as well as sticker 9. This extracted 

surface as defined by lOr, as well as the tracked location of each reflective object within that 
surface is then passed to composition of multiple overlapping images into individual full view 
frames unit 65. Unit 65 then catalogues into one complete set all detected reflective objects 
such as 3b, 4b, 7m and 9a and their corresponding extracted object pixels that have been 

10 detected over the entire field of view. As a matter of practice, the extracted pixels will be 
expressed as the minimum-bounding rectangle as depicted by lOr rather than a list of pixel 
coordinates. Within this minimum bounding rectangle, all background pixels will have been set 
to a null value by extraction unit 64 to clearly differentiate them from the foreground object 
during later examination. A center of gravity as well as polar coordinates to that center point 

15 are also calculated by Extraction Unit 64 and passed to composition unit 65. And finally unit 
64 will determine a beginning edge point to be associated with each object that can be used by- 
later routines to quickly trace the outline of the object from within its minimum bounding 
rectangle to perform a final extraction. It is also the responsibility of composition unit 65 to 
piece together reflective surfaces such as 3b, 4b, 7m and 9a as well as the objects to which 

20 they are attached such as pads 6 or helmet 8 or player 10 which may overlap frames taken by 
separate filming 25 or tracking cameras 24. 

Once the composition unit 65 has created the known set of polar coordinates to 
the centers of gravity for all know reflective markings and their corresponding objects and has 
also defined the minimum bounding rectangles and a beginning edge point this entire set of 

25 data is passed to the storage of tracking and video frames indexed by object and frame no. / 

time sequence unit 67. As the entire set of frames across the field of view created by the array 
of overhead assemblies 20c and perspective sets 30 are continually input by capture units 61 
they will be passed to tracking unit 62 which will follow each new object as it enters the field 
of view 20cl and ultimately leaves via the same assembly 20cl. The present inventors 

30 anticipate that after identifying the first appearance of an inked surface or corresponding object 
it may be more efficient to allow tracking unit 62 to express subsequent centers of gravity 
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using the same polar coordinate method except that the center point is no longer the center of 
Ice 2a but rather the previous center of gravity for the same inked surface or corresponding 
object. This changing center of reference strategy is shown in Fig. 4c by the sequence lOal 
through 10a4. 

5 Storage unit 67 will take in this continual flow of inked surface and 

corresponding object information and will create several databases for later analysis. First, 
each individual surface and object will have its own movements catalogued from the point it 
enters the field of view until at some later point it leaves. Note that the field of view is 
extended to cover team boxes 2f and 2g as well as the penalty box 2h. Hence, to system 100, 
10 there is no difference to tracking these surfaces and objects in these areas, where they are not 
active in the current event, as opposed to on the ice surface 2, where they are active, storage 
unit 67 is also responsible for creating and storing group information. For instance, unit 67 is 
expecting to find the following associations: 

1 . It will always true that a player 10 will have an associated right patch 7r 
15 and left patch 71. 

2. It will most often be true that a player 10 will also have an associated 
helmet 8 with sticker 9 and a stick 4 with tape 4a. 

3. It will sometimes be true that a player 10 will have puck 3 within its 
direct semi-circle of control. This semi-circle will be defined by the player 10's current 

20 location, direction, speed of travel and orientation and is depicted as 10s in Fig. 4c. 

4. It will sometimes be true that a player 10 may appear to have additional 
patches, helmets or sticks in which case multiple players may have collided. 

Whatever associations are detected are used to form a players group database. 
Note that storage unit 67 will distinguish between a stray stick or helmet and a player. As such, 
25 the stray stick or helmet will not create another instance of a player group. Hence, frame by 

frame a count will be made of to total number of unique player groups which should stay equal 
unless: 

1. A new player enters the field of view. 

2. An existing player exists the field of view. 
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3. There is a collision between one or more players such that their forms 
are temporarily joined. 

When two or more player groups are joined to form a single group somewhere 
within the field of view, it is anticipated that eventually the players will separate. At this point 
5 the system which had assigned the groups movements to each know player who had entered 
the group, will now begin to track individual players and the total group count will have 
returned to its pre-collision count. 

As storage unit 67 continually updates its various databases, this same 
information then becomes available in real time to quantification and analysis of object 
10 movement stored in cross-linked database unit 68. Quantification and analysis unit 68 will have 
a variable set of tasks it can perform on the tracking databases which can be enabled as either 
real-time or post event. In either case, some of those tasks are listed as follows: 

1. Correlate individual player group data to a pre-known list of potential 
players. This is accomplished by comparing the team indicia 7r2 as well as the player indicia 

15 712 to a pre-known database of values. 

2. Determine player orientation based upon the location of inked surfaces 
within the extracted objects in combination with the last known direction, speed and orientation 
of both the inked surfaces and their associated objects. 

Hence unit 68 will be able to identify that a player is skating backward versus forwards, or that 
their head and shoulders are beginning to turn and therefore it is expected that so will their 
direction of travel in the next captured frame. It should be further noted that reflective sticker 9 
was specifically designed to also assist unit 68 in determining whether the associated helmet 8 
(and therefore players head) is looking down or straight ahead. This is facilitated by the 
alternating black and white squares of sticker 9 that will vary in count depending upon helmet 
8's orientation. 

3. Update player statistics such as time entered and left ice surface (known 
as a shift), duration of shift, average speed, percentage of shift spent in defensive, neutral or 
offensive zones, percentage of puck control, number of shots, turnovers, steels, passes, etc. 
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4. Update team statistics such as number of shifts, average duration of 
shifts, percentage of play in defensive, neutral and offensive zones, percentage of puck control, 
number of shots, turnovers, steels, passes, etc. 

Any and all of this information that was pre-enabled to be tracked in real time 
is now available for inquiry and display on team output stations 80. After the event has 
completed, or during scheduled breaks in event activity, quantification and analysis unit 68 
may continue to process the stored databases in order to derive additional information which 
was not pre-enabled for real time. Other events, whether they are sporting or not and 
irrespective of the inclusion of human objects, will have their own set of uniquely desirable 
quantification and analysis requirements. 

At any time after the completion of the event and all of its analysis by unit 68, 
it may be desirable for selected individuals to remotely inquire upon the information tracked 
and analyzed by local computer system 60. Via the use of conventional remote 
communications devices depicted as 69 and 79 in fig. 8, a remote computer system for 
reviewing captured events depicted as 70 may be used to access the event databases. The 
present inventors do consider that the disclosed technique of first tracking specially inked 
surfaces which have been attached to objects which are desired to track and then extracting the 
minimum bounding areas of those objects movements is novel for at least the following 
reasons: 

1 . The technique operates in a niinimum of time and requires a minimum 
of computer processing by using a selected and narrow band of energy to illuminate and 
capture a small amount of information which yields object identity, orientation, direction of 
travel and speed at a minimum, irrespective of the ambient background reflection. 

2. The technique employs the quickly determined information from the 
narrow band of energy to efficiently extract a larger set of information from the entire band of 
energy typically found in ambient light. 

3. The technique creates a resultant movement database which can be used 
to reconstruct one or more portions or all of the event after it has concluded by overlaying all 
captured and extracted object movements back onto a pre-known ice surface background. 
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This last point is especially beneficial since in practice it allows what otherwise 
would be a prohibitive amount of event video information to be saved as a minimum set of 
movement information which can then be transferred via normal system to system connections 
such as the internet. Thus several individuals involved with an event can separately and easily 
5 download this data to their respective remote systems 70 for there own selective replay and 

analysis. It is even envisioned that only the centers of gravity need to be transmitted as opposed 
to all object pixels in the minimum bounding box since in a arcade style event replay the 
graphics of a players body, helmet and stick as well as the puck can easily be generated by the 
local system. 

10 An additional novel aspect of system 100 is depicted in Fig. 6. Specifically, by 

the use of pre-designated marked objects such as cone 11 with reflective ink 11a, it is possible 
to restrict system 100's field of tracking to some sub-portion of its field of view such as 2t. In 
practice an operator such as an on-ice coach will initiate this reduced tracking mode of system 
100 using a special entry code input to tracking control device 90. Device 90 transmits this 

15 control sequence via conventional RF to local computer system 60. This information is then 
provided to composition of multiple overlapping unit 65 that then searches for two or more 
pre-designated objects, e.g. cones 11, on the ice surface 2. When taken together, these two or 
more cones 11 will prescribe one or more geometric areas such as 2t within which all tracking 
is enabled and outside of which all tracking is disabled. In those cases where the cones have 

20 been placed in an ambiguous fashion, unit 65 will transmit for display all possible layouts 

given the current cones 11 detected from which the operator of control device 90 will select the 
appropriate layout. For example, the coach may simply place two cones 11 on Ice 2, one at 
either side of the red line that divides the ice surface into two parts. At this point unit 65 will 
display two possible layouts covering either one side of the ice or the other. 

25 After restricting system 100's tracking area, all other functionality is identical 

as previously described. Furthermore, quantification and analysis unit 68 is capable of 
outputting all calculated information to portable tracking control device 90 in response to the 
on-ice coaches real time inquiries. Unit 68 can be enabled by device 90 to also output the speed 
of successive shots on goal to a shot clock 91. 
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Thus the reader will see that the Multiple Object Tracking System provides a 
novel apparatus and method for: 

1- tracking multiple, fast moving, variably shaped objects such as humans 
in real time without the aid of operator intervention to either initially identify or conditionally 

5 re-identify the objects during the tracking period; 

2- reducing the time required to process image data by first separately 
tracking a special indicia which has been attached to the object after which the indicia's 
location may then be used along with the vectors of direction, acceleration and velocity to 
efficiently extract the object from its background; 

! 0 3- performing its recognition tasks independent of color distinctions within 

the objects; 

4- performing its recognition tasks independent of either passive or active 
electronic tracking devices; 

5- determining orientation information about the objects as a whole and 
1 5 potentially of their individual parts; 

6- performing under various lighting and air moisture conditions which 
may not be optimal for recognition techniques based upon visible light; 

7- easily constricting the field of tracking within the field of view; 

8- creating and maintaining in real time a movement database accessible 
20 by either time and or object(s) identity; 

9- outputting either all or some of the captured data either immediately 
within the ice arena of after the event to one or more remote computers over a communication 
network such as the Internet; and 

10- providing information to remote computer systems over the Internet that 
25 may be used for controlled event replay as well as critical analysis. 

While the above description contains many specifications, these should not be 
construed as limitations on the scope of the invention, but rather as an exemplification of 
preferred embodiments thereof. Many aspects of the systems functionality are beneficial by 
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themselves without other aspects being present. For instance, the separate filming cameras 
could be dropped as a matter of cost savings and / or speed improvement. The system would 
still then provide significant and novel benefit by tracking the inked surfaces attached to the 
objects to be tracked. There does not have to be an array of tracking cameras that are mounted 
5 directly above the event be tracked. If this arrangement proves to inconvenient, then the 

cameras simply need to be placed so that together they can create an overlapping field of view 
that is most likely to keep the ink surfaces in view. For instance, on the set of a movie 
production, this system could be used to automatically follow one or more aspects of the scene 
as the action unfolds. Hence, if the actors were to wear large marking made of the special ink 
10 on patches hidden underneath their clothes, than this tracking system could follow the actors 
movements and automatically direct the pan, tilt and zoom of selected production cameras. 

Another example of an aspect of the system that is beneficial but not mandatory 
is the link to a remote computer system for reviewing the captured events. While die present 
inventors do consider this system to be unique in the way it stores information which is 
15 especially beneficial for remote download, the remote system is not necessary for the system to 
have novelty and usefulness. 

It is evident from the description of the multiple object tracking system that it 
has applicability beyond that of tracking the movements of hockey players and the puck during 
an ice hockey game. For example, this same system could be set up over an outside roller 

20 hockey rink if the frame which holds the overhead assemblies where it self to be mounted on 
posts or polls to hold it above the playing area. The system could also be used to track 
basketball in a fashion very similar to ice hockey since these games are nearly always played in 
an indoor arena. Similar approaches could be used with other sports such as football and 
baseball as long as the field of view is sufficiently covered with perspective tracking cameras 

25 because there will not be any overhead assemblies. The system could also be used in large 
convention halls or auditoriums to track security whereabouts, attendee flow of support staff. 
This could be accomplished by using the same overhead tracking cameras while the filming 
cameras would more than likely be unnecessary. Each type of person to be tracked could be 
asked to where a special patch that could even be coded based upon statistically relevant 

30 criteria as determined by the event hosts. As individuals with patches moved about and visited 
different booths, their choices could automatically be tracked including their time spent at each 
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selected both. Such a system could also help with crowd flow if large lines where detected as 
forming around selected areas. Note that in this application, it is less critical that each any 
every movement of each and every person to be tracked is followed, but rather that in total the 
majority of movements of all like individuals are determined from which helpful decisions and 
5 statistics might be derived. 

From the foregoing detailed description of the present invention, the Multiple 
Object Tracking System, it will be apparent that the invention has a number of advantages, 
some of which have been described above and others of which are inherent in the invention. 
Also, it will be apparent that modifications can be made to the Multiple Object Tracking 
10 System without departing from the teachings of the invention. Accordingly, the scope of the 
invention is only to be limited as necessitated by the accompanying claims. 
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In the Claims 

We claim: 

1. An automated system for tracking the movement of multiple objects 
within a predefined area comprising: 

5 means for attaching special markings onto said objects where 

said markings are selected to be reflective to a selected narrow band of 
energy, 

means for radiating said narrow band of energy throughout 
said predefined area, 

10 means for receiving reflections of said radiated energy off of 

said special markings, and 

means for determining said location of said special markings 
and therefore said objects based upon said reflections. 

2. An automated system for tracking the movement of multiple objects 
15 within a predefined area comprising: 

means for attaching special markings onto said objects, 

means for tracking the location of said special markings, and 

means for employing said location of said special markings 
to extract the image of said objects from said object's background. 

20 3. An automated system for tracking the movement of multiple objects 

within a predefined area comprising: 

means for establishing a field of view which encompasses 
said predefined area, 

means for first minimally searching a portion of said field of 
25 view where said objects are expected to enter and exit said area, 

means for establishing said objects first position within said 
field of view based upon said first minimally searching, and 
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means for subsequently minimizing the field of view which 
must be searched to find said objects next location based upon said objects 
first and subsequently prior know position, it last calculated direction 
vector and its pre-known maximum rate of travel. 

4. An automated system for remotely providing selective information 
regarding the movements of multiple objects within a predefined area comprising: 

means for establishing a field of view which encompasses 
said predefined area, 

means for marking each of said objects with a unique 

indicia, 

means for tracking said indicia, 

means for establishing a minimum database of the 
movements of said unique indicia cross-linked to said marked objects, and 

means for selectively transmitting some or all of said cross- 
linked database to a remote system for subsequent analysis. 

5 . An automated system for tracking the movements including direction 
and orientation of multiple objects within a predefined area comprising: 

means for establishing a field of view which encompasses 
said predefined area, 

means for marking each of said objects with a unique 

indicia, 

means for tracking said direction and said orientation of said 
unique indicia, and 

means for determining said direction and said orientation of 
said objects based upon said direction and said orientation of said unique 
indicia. 

6. A method for efficiently extracting an object from its background image 
as it changes its position from frame to frame comprising the steps of: 



WO 00/31560 



PCTYUS99/27281 



-35- 

marking said object with a special frequency selective 
reflective material, 

radiating upon said object an energy whose frequency 
matches the reflective properties of said material, 

receiving reflections of said radiated energy off of said 
material through a camera fitted with a filter designed to pass the 
frequencies matching said radiated energy, 

determining the location of said reflective material by 
searching images captured by said filtered camera, 

capturing time and location matching images of said object 
with an urifiltered camera, and 

efficiently extracting said object from said matching images 
based upon said determined location of said reflective material. 
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Multiple Object Tracking System 

Field of Invention 

The present invention relates to systems for tracking the movement of multiple 
objects within a predefined area. 

5 Background of the Invention 

As camera, micro-electronic and computer system technology continues to 
advance at a great pace there has been an increasing supply of machine vision systems intended 
to replace well defined, repetitive vision / recognition tasks which had previously been 
performed by humans. Early systems were designed to recognize parts moving along assembly 

10 lines to aid the manufacturing process. More recently, many inventions have been put forth to 
recognize humans and there movement. The variability of humans and their clothing as well as 
the complexity of the backgrounds within which they endeavor has presented a significant 
challenge to state-of-the-art technology. Considerable attention has been paid to various 
techniques for discerning the human shape from its background using edge detection 

15 techniques that look to remove stationary, i.e. background, information. There are two major 
factors that effect the success of these various techniques. The first is image resolution, which 
drives the amount of information and hence detail that is available to the accompanying 
computer system in order to differentiate foreground from background. Ideally the greater the 
resolution the better. However as resolution increasing so does the cost of the camera as well 

20 as the accompanying computer system. Even more importantly, as resolution increases time to 
process increases significantly. And as time to process increases the ability for these systems to 
perform real-time operations is impeded. 

The following are seven examples of machine vision systems designed in some 
way or another to recognize human or object motion from within a predefined area. 

25 In November of 1994 U.S. Patent 5,363,297 entitled Automated Camera-Based 

Tracking System for Sports Contests issued to Larson et al. This system employed multiple 
cameras to continuously monitor the area of an ongoing sporting event. Each camera would 
feed its information to an accompanying computer system for analysis that consisted of 
extracting the players from the stationary background and thereby tracking their silhouettes. 
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The inventor had anticipated problems when the individual players would collide or otherwise 
engage each other and hence merging their individual silhouettes. A need was also recognized 
to initialize the system by first identifying each player as they appeared within the system's 
field of view. Larson et al. specified two solutions for these problems. First, they proposed 
5 attaching monitors to the tracking system that would be operated by humans who would 

perform the initial recognition as well as all subsequent re-identifications when the system lost 
track of a player due to a merging of silhouettes. Secondly, they proposed attaching electronic 
tracking devices and employing triangulation of received signals in order to identify and track 
individual players. There are at least four major problems with the Larson patent. First, the 

10 amount of digital processing required to perform the player extraction in real time greatly 

exceeds the cost-effective computer technology of today, let alone 1994. Secondly, in order to 
perform this extraction, a greater amount of detail would be necessary which would thereby 
increase the cost of implementation by requiring more cameras and related computer systems. 
And of course the additional detail would only tend to further slow down the responsiveness of 

15 the system. Thirdly, the requirement of one or more operators to initially recognize and then 
re-identify players is extremely limiting and costly. This requirement essentially made the 
patent economically impractical for monitoring non-professional youth sporting events, where 
the system cost including the ongoing cost of the human operator would greatly exceed smaller 
revenue streams. It should be noted that this operator would more than likely be a parent of 

20 one of the youths who would probably be unfamiliar with all of the players and who would 

more than likely find it stressful to make so many decisions in real time. This approach would 
also require training and retraining operators, which would also be prohibitive. Fourthly, the 
type of electronics necessary to track players in real time would have to operate at higher 
frequencies that would also mean that it would be more expensive, providing a further 

25 economic drawback. The fifth major problem is that the system could have a difficult time 

determining the orientation of the player. For example, while Larson's invention could detect 
the direction of an individual's motion, it could not determine if the person was facing 
forwards or backwards, let alone if the person's head was turning in another direction. 

In December of 1995, U.S. patent number 5,473,369 entitled Object Tracking 
30 Apparatus issued to Keiko Abe. This system was concerned with the actual image processing 
techniques used to follow an object from frame to frame. The inventor described prior an that 
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compared the images block by block from one frame to the next, where a block is assumed to 
be one or more pixels of the image. It was pointed out that such systems depended upon time 
consuming and error prone statistical calculations that were especially susceptible to 
misinterpretation when the object changed size within the field of view or disappeared 
5 altogether. Abe proposed taking the same video frames but first separating them into luminance 
and color histograms that are then to be compared frame by frame. By comparing the 
histograms rather than blocks Abe argued that the system would be more accurate and efficient 
that the block matching systems. However, there are at least five major problems with Abe's 
patent. First, the effectiveness and reliability of this technique is highly dependent upon the 

10 lighting conditions initially and over time within the field of view being tracked. For instance, 
if the initial frame was taken under a well-lit condition, the luminance histograms of the object 
may be ideal. However, when the lighting conditions are poor to begin with, or worse yet 
change from frame to frame as might happen with sudden burst of ambient light, the luminance 
histograms will be subject to considerable error. Secondly, relying upon color histograms is 

15 equally uncertain due partly to the susceptibility of color detection to lighting conditions, which 
again may vary from frame to frame, and in part to the potential for object and background 
blurring when color schemes overlap. A third problem is that Abe's system does not lend itself 
to tracking multiple objects that may have similar or identical luminance / color information 
and which may overlap from frame to frame. A forth problem is discussed in Abe's 

20 specification that indicates a requirement of a human operator to initialize the system by 

selecting the portion of the video image which contains the object to be tracked, his so-called 
region designating frame. This requirement would be even more restrictive when consideration 
is given to multiple object tracking where the objects may go in and out of the field of view of 
temporarily overlap each other. And finally, a fifth problem is alluded to in the specification 

25 where it is expressed as an opportunity of the system to automatically control the pan, tilt and 
zoom of a camera. In so doing, Abe states that the system is capable of "coping with any 
change in the in the size of the object and which can photograph the target object always in a 
desirable size, thereby attaining a substantial improvement in terms of the facility with which 
the apparatus can be used." Hence, it is recognized that this method / apparatus is still very 

30 resolution dependent similar to the block methods it is attempting to improve upon. 
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In April of 1997, U.S. Patent Number 5,617,335 entitled System for and 
Method of Recognizing and Tracking Target Mark issued to Hashima et al. This invention is 
attempting to address the problem of determining the three-dimensional coordinates of an 
object with respect to a tracking camera and processing mechanism, e.g. a robotic arm from a 
5 single two-dimensional image. He expresses these coordinates as the attitude and position of a 
target mark that has been placed upon the object to be tracked. In Hashima's review of the 
prior art he lists several existing methods many of which require too many calculations and or 
have problems with multiple objects and background image noise. He discloses a technique for 
marking the object to be tracked with a white triangle inside of a black circle. Once the special 

10 markings are captured, they are quickly converted into projected histograms in the X and Y 
directions of the image of the triangle mark after which the centers of gravity as well as the 
maximum histogram values in the X and Y directions are also determined. All of this 
information is then collectively used for "determining which of classified and preset attitude 
patterns the attitude of the triangle of said target mark belongs to based upon the position of the 

15 centers of gravity, the maximum histogram values, the X and Y-axis values, and the known 

geometrical data of said target mark". Even taking Hashima's assertion of increased efficiency 
and accuracy, his technique has at least three major limitations. First, the object to be tracked 
must be marked in a highly accurate fashion and this mark must be visible to the tracking 
camera at all times. No provision has been disclosed at to how the object can be tracked if the 

20 markings are temporarily blocked from the view of the tracking camera. Secondly, by 

attempting to determine three dimension information from a single two dimensional image, 
Hashima is focusing his solution on situations were additional perspective cameras may not be 
available. Given such additional cameras, there exist even more efficient and accurate methods 
for determining the third dimension. Thirdly, this invention teaches of a system which 

25 functions well "even when an image either contains many objects or has a lot of noises". 

However, if every one of these multiple objects needed to be tracked within the same image, 
Hashima's invention would not be at optimal performance since at any given time the preferred 
orientation of camera to object cannot be simultaneously maintained for multiple objects 
scattered in three dimensions. 

30 In March of 1998, U.S. Patent Number 5,731,785 entitled System and Method 

for Locating Objects Including an Inhibiting Feature issued to Lemelson et al. This invention 
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teaches the tracking of objects by "an electronic code generating system or device carried by 
the object in a portable housing". This "system or device" is specified to receive locating 
signals such as from the GPS constellation or a ground based triangulation setup. It then uses 
these signals to determine it's own location. Lemelson anticipates that at some point in time 

5 operators of a remote tracking system may be interested in the exact whereabouts of one 
individual object from amongst the multiplicity of objects that are housing such tracking 
devices. In order to determine the objects location, the tracking system will first transmit a 
unique "inquiry signal" coded for one particular device in one particular object. AH of the 
individual tracking devices will then receive this signal but only the one device whose 

10 identifying code matches the "inquiry signal" will respond. This response is in the form of a 
transmission that includes the tracking devices currently determined location. The tracking 
system then receives this signal and displays on a computer system monitor related information 
about the identified / located object. Lemelson et al.'s invention is primarily applicable to the 
tracking of many objects over a very wide area, so wide that these objects are out of range of 

15 any reasonably sized camera tracking system. As an apparatus and method for tracking objects 
that are within a range suitable for a camera network, this invention has at least three major 
problems. First, it requires that each object have the capability to constantly monitor and track 
its own location. Such a requirement involves the use of a computing device which must be set 
up to receive GPS or other tracking signals and also transmit locating signals. Such a device 

20 will typically take up more space than a marking or indicia that can be placed on an object and 
then tracked with a camera. Furthermore, this device will require power. Secondly, Lemelson 
et al.'s invention assumes that the remote tracking station is only interested in one or a smaller 
fraction of all potential objects at a given time. However, there are many situations when it is 
desirous to follow the exact and continuous movements of all tracked objects as they move 

25 about within a predefined area. Whereas it is conceivable that this invention could constantly 
transmit "inquiry signals" for all objects and constantly receive locating signal responses, it is 
anticipated that this amount of information would unacceptably limit the movement resolution 
of the system. Thirdly, such an electronic based system has no understanding of an object's 
orientation with respect to it's direction of movement. Hence, while it is possible to determine 

30 the direction a car or person being tracked is moving, it is not shown how the system could 
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determine if that same car or person was facing or turned away from it's current direction of 
travel. 

In June of 1998, U.S. Patent No. 5,768,151 entitled System for Determining 
the Trajectory of an Object in a Sports Simulator issued to Lowery et al. This invention teaches 
the use of stereoscopic cameras trained upon a limited field of view to follow the trajectory of 
an anticipated object. As the object transverses the field of view the cameras capture images at 
a suitably slower rate such that the object creates a blur as it moves. This blurred path is then 
analyzed and converted into the object's trajectory vectors within the field of view. Another 
key means of Lowery et al. 's apparatus is its ability to determine when it should begin 
tracking. As such a sound detecting device is specified to sense the presence of the object 
within the field of view after which the image capture system is then immediately activated. 
There are at least four major limitations with Lowery et al. 's invention that would hinder its 
broader applicability. First, this invention expects a very narrow range of motion of the object 
and as such has a significantly restricted field of view. If the concept were to be applied to a 
larger area then multiple perspective cameras would need to be employed. The system would 
then also need to determine which cameras should be activated once the object is detected to be 
traveling within the tracking region. However, without actually first determining where the 
object is located, say for instance by attempting to triangulate the sound emitted by the object, 
the system would have no idea which cameras to activate. Hence all cameras would need to 
capture images creating a very large set of data that would need to be parsed by the tracking 
computer in order to determine the location of the object. The second limitation is that this 
system does not attempt to uniquely identify each object that it senses. Hence, while it is 
capable of determining a trajectory vector for individual objects, it does not anticipate a need 
for, nor disclose a method of determining the unique identity of each object as it passes by. 
The third limitation deals with the ability to track multiple objects simultaneously within this 
same field of view. Since this invention anticipates only one object at a time, it is merely 
detennining trajectory vectors, and not the object's identity. Hence if two or more objects are 
traveling throughout the tracking region and they collide in such a way as to effect each other's 
path of travel, then the system will be left to determine which object continued on which path 
after the merge event. The forth limitation deals with the systems inability to pick up the object 
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from its background when there is insufficient color and or luminescence difference between 
the two. 

All of the above listed prior art in one way or another was attempting to track 
the movement of at least one object within a predefined area. When taken in combination, their 
5 limitations that must be overcome in total are as follows: 

1- If the tracking system attempts to differentiate between the object and 
its background purely on the basis of pixel by pixel comparison as does Larson et ah, then the 
video image must have higher resolution to be accurate and the resulting computer processing 
time prohibits real time operation. 

10 2- If the tracking system attempts to reduce processing time by performing 

averaging techniques based upon separated color and luminescence information as does Abe, 
then accuracy is compromised especially as colors merge between multiple objects and their 
background or lighting conditions fluctuate substantially between image frames. Such reduction 
techniques are further hampered as object size diminishes which essentially reduces the amount 

15 of object versus background information thereby increasing "noise". The only solution is to 
zoom in on the object being tracking to keep the proper ratio of object to background 
information. This then implies that each object being tracked must have its own camera thereby 
greatly reducing the effectiveness of these techniques for tracking either more objects and / or 
greater fields of view. 

20 3- If the tracking system such as Lowery et al.'s employs two perspective 

cameras and an image blurring technique to capture three-dimensional trajectory information it 
reduces image processing requirements but looses important video image detail. 

4- If the tracking system such as Hashima et al. 's employs detailed indicia 
placed upon the object to be tracked, this can be effective to reduce the amount of image 
25 processing. However, Hashima faces significant issues when trying to determine three- 
dimensional information from a single two-dimensional image, which is one of his mandatory 
requirements. His resultant techniques preclude the tracking of multiple fast moving objects 
over wider fields of view where the object's indicia may at times be blocked from view or at 
least be at significantly changing perspectives from the tracking camera. 
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5- All of the video / camera only based techniques such as Larson, Abe, 
Hashima and Lowery are prone to error if they were to track multiple objects whose paths 
would intersect and / or collide. Only Larson specifically anticipates this type of multiple 
object tracking and suggests the use of human operators to resolve object overlap. Such 

5 operators are cost prohibitive and also limited in their capacity to keep up with multiple fast 

moving objects in real time. While, as Larson suggests, it is possible to use passive electronics 
to help identify objects once the system determines their identities have been lost, these devices 
will have their own resolution / speed of response restrictions which are cost sensitive. 

6- Furthermore, both Larson and Abe's video / camera solutions anticipate 
10 the requirement of a human operator to initialize the system. Larson would require the operator 

to identify each object for the system. These objects would then be automatically tracked until 
they merged in some way with another object at which time the operator would be needed to 
re-initialize the tracking system. Abe would require the operator to crop the initial image down 
to a "region designating frame" which essentially reduces the processing requirements to at 
15 least find if not also track the object. The intervention of any operator is both cost prohibitive 
and real-time-response limiting. 

7- Lowery 's video / camera solution anticipates automatic tracking 
activation based upon the sound detected presence of an object within the field of view. This 
technique is inherently limited to objects that make distinguishing sounds. It is also unable to 

20 track multiple objects that might be making similar noises simultaneously within the given field 
of view. 

8- If the tracking system attempts to eliminate image processing 
requirements by employing active electronic tracking devices such as Lemelson et al., then the 
objects are required to house powered devices capable of receiving and processing both 

25 locating and inquiry signals. Such devices limit the range and type of objects that can be 

tracked based upon the practicality and cost of embedding computing devices. In systems such 
as Lemelson's that employ electronics alone, the wealth of information available from image 
data is lost. Furthermore, such systems may be able to track location but cannot track object 
orientation, e.g. is the object traveling forwards or backwards. 
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9- With the exception of Hashima's indicia technique, all of these solutions 
are still not capturing object orientation information. Such information can be extremely 
important to anticipate future object movement. 

10- All of the video / camera based solutions will have difficulty picking up 
5 fast moving objects whose color and or luminescence information is sufficiently close to that of 

other tracked objects or the image background no matter what technique is employed. All non- 
video based solutions will give up valuable image information. 

Summary of the Invention 

While the present invention will be specified in reference to one particular 
10 example of multi-object tracking as will be described forthwith, this specification should not be 
construed as a limitation on the scope of the invention, but rather as an exemplification of 
preferred embodiments thereof. The inventors envision many related uses of the apparatus and 
methods herein disclosed only some of which will be mentioned in the conclusion to this 
applications specification. For purposes of teaching the novel aspects of this invention, the 
15 example of multi-object tracking is that of a sporting event such as hockey. The particular 
aspects of hockey which make it a difficult series of events to track and therefore a good 
example of the strengths of the present invention over the prior art are as follows: 

1- There are no other human based activities know to the present inventors 
where the humans as objects can travel at a faster speed or change directions and orientation 
20 more quickly than hockey. On skates a player's speed can approach twenty-five miles per 
hour, which is considerably faster than any activity involving walking or running that is still 
conducted on the ground without the aid of a vehicle of some sort. Tracking these faster 
movements, especially given the variability of the human form, challenges the real time 
performance aspects of the system. 

25 2- The speed of the object being contested by the players, i.e. the puck, 

can travel at rates of up to one hundred miles per hour and is also subject to sudden and quick 
changes of direction. This combination of speed and re-direction presents a difficult tracking 
problem and is unique in athletics. Tracking the puck is easier than tracking player when 
considering that the puck will not change shape yet the puck travels at roughly four times the 

30 speed, is on the magnitude of one hundred times as small and may travel in three dimensions. 
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S' The individual players are constantly entering and exiting the tracking 
field of view and as such must be efficiently and automatically identified by the tracking system 
in order for real time performance. 

4- While in the field of view, both the puck and the players are routinely 
either fully or partially hidden from view as they merge with the paths of other players. This 
creates a challenge to follow movements with often limited or no image data. 

5- The lighting conditions are difficult to work with since the ice surface 
will create a highly reflective background that could tend to saturate the CCD elements of the 
camera while the area itself may be subject to sudden bursts of light from either spectator's 
camera flashes or in-house lighting systems. This places limitations on luminescence based 
tracking techniques. 

6- The colors of the players on the same team are identical and may often 
match the markings on the ice surface and surrounding rink boards. This places limitations on 
color based tracking techniques. 

7- It is not unusual for a hockey game to be played while a certain level of 
fog exists within the arena. This challenges any camera-based system since it could greatly 
reduce visibility of the players and puck. 

8- Hockey is a filmed event and as such it presents the opportunity not 
only to track the movement of multiple objects but also to determine a center of interest which 
is constandy and abruptly changing. Once this center is determined, there is a further 
advantage to automatically direct the tilt, pan and zoom of a broadcaster's camera to follow the 
action from a perspective view in real time. Automatically directing a camera that is situated 
for a perspective view presents a difficult problem for a machine vision system since it is 
considerably harder to follow the objects in three dimensions in real time as a perspective view 
would require. 

9- Each individual player as well as the coaches in a game may at any time 
be instantly desirous of obtaining information regarding themselves, a group of players, their 
entire team and / or the other team. This information may furthermore pertain to the entire or 
some sub-set of the duration of the activities from start to present. Such requirements place a 
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demand on the tracking system to quickly and efficiently store information in a way that it may 
easily be recalled from many viewpoints. 

10- The enclosed metal and cement block arena precludes the use of GPS 
and presents difficulties for the use of passive electronic tracking devices due to the many 
potential reflections of in-house triangulation signals. The players themselves and the nature of 
the game and its potential for significant high impact collisions limit the desirability of placing 
active electronic devices within their equipment. Since these devices must carry a power source 
they will in practice take up enough space to present a potential hazard to the players. 
Furthermore, such devices would be extremely cost prohibitive at the local rink level where 
literally hundreds of children are playing games every week and would each need their own 
devices or to share devices. 

11- Player orientation and location of limbs with respect to body are very 
important information to track. A player may be traveling forward facing forward while they 
are turning their head to the side to view the development of play. The turn of the head 
becomes important information for coaching analysis since it is an indication of a player's on- 
ice awareness. Furthermore, a play may almost instantly rotate their orientation while still 
proceeding in the same direction such that they are now traveling backwards instead of 
forwards. This is also very critical information. A player's head may be tilted down towards 
the ice, which is not desirable for prolonged periods of time or if often repeated. All of this 
information is important to track but presents problems over and above simply tracking the 
player as a whole. 

12- Limiting the size of the area of tracking is desirable during practice 
sessions where individual drills may be conducted on a limited portion of the ice with a small 
number of players at a time. Under these conditions it would be desirable to easily restrict the 
system's tracking area within its field of view. 

13- The number and speed of player changes and collisions is so great that 
utilizing human intervention to identify and re-identify players would be significantly stressful 
and error prone, if not practically impossible, especially at the local rink level. 

Given the current state of the art in camera systems, non- visible energy sources 
and filters, digital image processing and automated camera controls it is possible to create an 
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entirely automated multi-object tracking system which operates within a predefined area and 
tracks the constant location, orientation and direction of movement of each and every object 
within the field of view. Such a system greatly increases the ability of the participants and 
observers to understand, analyze and enjoy the given activity. 

Accordingly, the objects and advantages of the present invention are to provide 
a system for tracking multiple objects within a predefined area with the following capabilities: 

1- to provide a system for tracking multiple, fast moving, variably shaped 
objects such as humans in real time without the aid of operator intervention to either initially 
identify or conditionally re-identify the objects during the tracking period; 

2- to reduce the time required to process image data by first separately 
tracking a special indicia which has been attached to the object after which the indicia's 
location may then be used along with the vectors of direction, acceleration and velocity to 
efficiently extract the object from its background; 

3- to provide a system which is not dependent upon color distinctions 
within the objects; 

4- to provide a system which does not rely upon either passive or active 
electronic tracking devices; 

5- to provide a system which determines orientation information about the 
objects as a whole and potentially of their individual parts; 

6- to provide a system which can work under various lighting and air 
moisture conditions which may not be optimal for recognition techniques based upon visible 
light; 

7- to provide a system where the field of tracking is easily constricted 
within the systems field of view; 

8- to provide a system which can create and maintain in real time a 
movement database accessible by either time and or object(s) identity; 
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9- to provide a system which can output either all or some of the captured 
data either immediately within the ice arena of after the event to one or more remote computers 
over a communication network such as the Internet; and 

10- to provide a system such that the information output to remote computer 
systems over remote communications such as the Internet may be used for controlled event 
replay as well as critical analysis. 

Further objects and advantages are to provide a cost efficient system to build, 
install and maintain with a minimum of moving parts that is capable of operating under a range 
of temperature conditions. Still further objects and advantages of the present invention will 
become apparent from a consideration of the drawings and ensuing description. 

Description of the Drawings 

Fig. 1 is a top view drawing of the preferred embodiment of the present 
invention depicting an array of overhead X-Y tracking cameras which when taken together 
form a field of view encompassing the skating and bench area within an ice hockey arena. Also 
depicted are perspective Z tracking camera sets behind each goal, automatic pan, tilt and zoom 
perspective filming cameras as well as a single representative player and puck. 

Fig. 2a is a set of three perspective drawings depicting a typical player's jersey, 
pads with tracking patches in place and then a combination of the jersey over the pad with 
patches. 

Fig. 2b is a set of two perspective drawings depicting a hockey puck as well as 
a typical player's hockey stick, where each has been augmented to include tracking ink on at 
least some portion of their outer surfaces. 

Fig. 2c is a set of two perspective drawings depicting a typical hockey players 
helmet which has been augmented to include tracking stickers on at least some top portion of 
its outer surface. 

Fig. 3a is a perspective drawing of a typical hockey players pads, helmet, stick 
and puck being captured from an overhead X-Y filming camera and displayed on a viewing 
screen. 
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Fig. 3b is a perspective drawing similar to Fig. 3a except that now trackina ink 
has been added to the hockey stick and puck, tracking patches have been added to the pads"and 
tracking stickers to the helmet. In addition, a tracking energy source as well as a frequency- 
matching filter has been added to the overhead X-Y fflrning camera making it a tracking 
camera. 

Fig. 4a is a perspective drawing similar to Fig. 3b except that now an 
additional un-filtered filming camera has been added to the overhead X-Y tracking camera so 
as to efficiently combine both sets of information. 

Fig. 4b is a top view depiction of a key element of the present invention's 
novel process for efficiently extracting the video image of the object being tracked by first 
locating an attached indicia and then working out away from the indicia comparing each pixel 
of the image to a pre-known background so as to effectively outline the tracked object. 

Fig. 4c is a top view of a portion of an ice arena showing a series of tracked 
and traced motions of a typical hockey player, stick and puck by the overhead X-Y filming and 
1 5 tracking cameras depicted in Fig. 4a. 

Fig. 5 is a perspective drawing of a mountable camera array frame with 
overhead X-Y filming and tracking cameras attached to the roof of a typical ice arena. 

Fig. 6 is a top view drawing of an ice arena where area restricting cones with 
tracking ink have been placed so as to indicate to the overhead X-Y tracking cameras that only 
a sub-portion of the entire field of view is to be tracked. Also depicted are typical players, one 
is within the tracking zone while many are outside of the zone. Further depicted is a portable 
device that may be used by on ice coaching staff to control the functions of and inquire upon 
the information generated by the present invention. 



20 



25 



Fig. 7 is a front view drawing of a hockey players bench which has been 
outfitted with a series of display devices with keypads that may be used by typical hockey 
players during the course of a hockey game to inquire upon the information generated by the 
tracking system. 

Fig. 8 is a block diagram depicting all of the tracking and computing elements 
of the proposed present invention. 
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Detailed Description of the Invention 

Referring to Fig. 1 there is shown a top view drawing of the preferred 
embodiment of the Multiple Object Tracking System 100. System 100 comprises an array of 
overhead x-y camera assemblies 20c that individually track all object movement within a fixed 
5 area such as 20v. In total, the array of overhead assemblies 20c track all movements in ice 

playing surface 2, team boxes 2f and 2g, penalty box 2h as well as a portion of entrance-way 
2e. Assembly 20c further comprise filming camera 25, energy source 23, tracking camera 24 
onto which is attached energy filter 24f , all of which is housed in assembly casing 21 and has a 
view to the ice surface 2 below through assembly Plexiglas 21a. Energy source 23 emits 

10 selected energy 23a which radiates down onto surface 2 and off of the objects moving upon 
this surface such as player 10 and puck 3. It should be noted that selected energy 23a was 
specifically chosen to be of a frequency that not only is outside of the range of normal ambient 
light but also will not be effected by moisture conditions such as fog. A specially situated 
overhead assembly 20cl is shown as covering ice surface 2's entrance-way 2e. This assembly 

15 20cl is identical in construction to 20c and is important and will later be specifically referenced 
because all objects to be tracked must enter and exit through this camera's field of view. Also 
tracking movements on a selected portion of ice surface 2 are perspective z tracking camera 
sets 30 that are situated as one pair at both ends of the ice 2. And finally there are automatic 
filming cameras 40 which are constantly being directed to the center of play as represented by 

20 player 10 who is currently controlling puck 3. Automatic filming cameras 40 are in continuous 
communications with and are receiving their directions from local computer system for video 
processing and analysis 60. System 60 itself is also in continuous communications with array 
of overhead x-y tracking camera assemblies 20c and perspective z tracking camera sets 30. 
Local system 60 is in further in optional communication with remote computer system for 

25 reviewing captured events 70 which has attached viewing monitor 27 that displays scene 28. 
Also in optional communication with local system 60 are team output stations 80 which 
provide information on the object movements tracked from the beginning of the event to the 
current moment. 

Referring now to Fig. 2a there is depicted a typical player's jersey 5 and 
30 player's shoulder pads 6. Affixed to pads 6 are right shoulder team patch 7r and left shoulder 
player patch 71. Patch 7r comprises orientation mark 7rl, which is an arrowhead pointing 
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away from the head towards the arm and team indicia 7r2, which is a unique bar code. Patch 
71 comprises orientation mark 711, which is also an arrowhead pointing away from the head 
towards the arm and player indicia 712, which is a unique number. It should be noted that the 
indicia on patches 7r and 71 are created from a frequency selective reflective material, 
preferably an ink. Also referring to Fig. 2a there is depicted jersey 5 placed over pads 6. Note 
that jersey 5 is also shown to be cut-away for a full view of underlying player patch 71. 

Also depicted in Fig. 2a is reflected energy 7m which is shown radiating 
though transmissive jersey 5. Such techniques for seeing through cloth are not new and were 
demonstrated by Sony Corporation when they placed a shirt over a license plate and then took 
two pictures, one without a filter and the other with a visible light filter. Since cloth reflects 
light in the visible spectrum so intensely, the plate could not be seen by the unfiltered image. 
But with the infrared pass filter only the infrared light was captured making the numbers and 
letters on the license plate visible. It should be noted that the present invention teaches the use 
of special frequency selective reflective material that will further enhance the contrast, i.e. 
signal to noise ratio, of the desired image above and beyond the filtering techniques discussed 
by Sony. 

Referring now to Fig. 2b there is shown a typical hockey puck 3 where its top 
surface (and in practice all outer surfaces) has been coated with a reflective ink 3a similar to 
the frequency selective reflective material used in patches 7r and 71. In response to a particular 
frequency of energy as would be emitted for example by energy source 23, ink 3a causes 
reflected energy 3b. Also shown is a typical hockey stick 4 where its blade has been wrapped 
with a special reflective hockey tape 4a that has been produced to include the same special 
reflective ink. In response to a particular frequency of energy as would be emitted for example 
by energy source 23, reflective tape 4a causes reflected energy 4b. 

Referring now to Fig. 2c there is shown both a top and perspective view of a 
typical hockey players helmet 8 where a reflective sticker 9 has been applied to its top surface 
and has been produced to include the same special reflective ink. In response to a particular 
frequency of energy as would be emitted for example by energy source 23, sticker 9 causes 
reflected energy 9a. 
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Referring now to Fig. 3a there is shown a first embodiment of the overhead x-y 
tracking camera assembly 20a. In this embodiment, assembly 20a has been limited to tracking 
camera 24 (without energy filter 24f) which is enclosed within assembly casing 21 and has a 
view to the ice surface 2 below through assembly Plexiglas 21a. There is depicted below 
5 assembly 20a unmarked player 10, unmarked stick 4 and unmarked puck 3. Also show is cable 
26 which attaches assembly 20a to local computer system 60 (not depicted), to remote 
computer 70 (also not depicted) and therefore to viewing monitor 27 that displays scene 28. 

Referring now to Fig. 3b there is shown a second embodiment of the overhead 
x-y tracking camera assembly 20b. In this embodiment, assembly 20b has been augmented to 

10 include energy source 23 that emits selected energy 23a which radiates down onto the objects 
moving upon the ice surface such as player 10, puck 3 and stick 4. Note that player 10's pads 
6 have been augmented to include right shoulder team patch 7r and left shoulder player patch 
71. Also note that Puck 3 now includes reflective ink 3a and that stick 4 has been wrapped with 
a special reflective hockey tape 4a. Scene 28 now depicts a much smaller set of information to 

1 5 be analyzed and tracked . 

Referring now to Fig. 4a there is shown a third and the preferred embodiment 
of the overhead x-y tracking camera assembly 20c. In this embodiment, assembly 20c has been 
augmented to include filming camera 25 that captures unfiltered images of all movement on the 
ice surface below. Fig. 4b depicts a key element of the present invention's novel process for 

20 efficiently extracting the video image of the object being tracked by first locating an attached 

indicia and then working out away from the indicia comparing each pixel of the image to a pre- 
known background 2r. Once it is determined that background pixels have been reached, the 
image-tracing path lOp continues around the object until a closed path is completed. Around 
This closed path lOp a minimum bounding rectangle lOr is calculated in order to quickly 

25 extract the portion of the video image containing the tracked object. Referring also to Fig. 4c 
there is depicted the processed motion of player 10 upon ice surface 2 using the preferred 
assembly 20c. Player 10 is shown to transverse through four movements points lOal, 10a2, 
10a3 and 10a4 carrying puck 3 along the way towards taking a shot at goal 2b within 
calculated shot triangle 2c. The center of ice surface 2 is shown as point 2a from which first 

30 movement angle lOalo and movement radius lOalr have been calculated to represent player 
10' s first movement to point lOal. Also calculated are movement angle 31o and movement 
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radius 31r that describe puck 3's initial movement to point lOal. About center point 2a there is 
shown a maximum bounding circle 10b within which an object's next location is calculated to 
be reside based upon the objects last known position (in this case point 2a) as well as its last 
calculated vectors of direction, acceleration and velocity. And finally there is depicted the 
5 maximum initialization search rectangle lOe which covers the outermost portion of the field of 
view as tracked by overhead assembly 20cl which is situated over the only entrance-way 2e to 
ice surface 2. 

Referring now to Fig, 5 there is shown overhead mounting frame 50 that is 
variably attachable to differing constructions of rink girders 2d. Further attached to frame 50 is 
10 the array of overhead x-y tracking camera assemblies such as 23c that emit selected energy 23a 
down upon the ice surface 2 below. Assemblies 23c are connected to local computer system 
for video processing / analysis 60. 

Referring now to Fig, 6 there is depicted ice surface 2 whose tracking area has 
been restricted by the placement of four typical cones 11 which have been coated with a 

15 reflective ink 11a. In response to a particular frequency of energy as would be emitted for 
example by energy source 23, ink 11a causes reflected energy lib. Shown within restricted 
region 2t is a single player 10 whose current motion is to be tracked while he remains within 
region 2t. Shown outside of region 2t are multiple players such as lOi whose movements are 
not being tracked. Also shown are shot clock 91 and portable tracking control device 90 that 

20 are both in optional communication with local computer system 60. 

Referring now to Fig. 7 there is shown a close up view of team output stations 
80 that are attached to team boxes 2f (shown) and 2g, as well as penalty box 2h. 

Referring now to Fig. 8 there is depicted a block diagram of the entire system. 
With respect to overhead x-y tracking camera assembly 30c there is shown additional optional 

25 rf modem 25m which may be used in place of wire 26 (not depicted) to link filming camera 25 
to local computer system 60. Also shown is additional optional RF modem 24m which may be 
used in place of wire 26 (not depicted) to link tracking camera 24 to local computer system 60. 
Likewise additions are also depicted for perspective z tracking camera sets 30. Local system 60 
has been broken down into its constituent block parts starting with the video capture for 

30 tracking camera(s) unit 61 which accepts input from all tracking camera(s) 24 mounted within 
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overhead assemblies 30c or perspective sets 30. Capture unit 61 then feeds its data stream to 
indicia / object tracking analysis unit 62. In time parallel operation there is shown video 
capture for filming camera(s) unit 63 which accepts input from all filming camera(s) 25 
mounted within overhead assemblies 30c or perspective sets 30. Capture unit 63 then feeds its 

5 data stream to object / scene extraction analysis unit 64. Unit 64 also receives simultaneous 
input from indicia / object tracking analysis unit 62. Both extraction analysis unit 64 and 
tracking analysis unit 62 simultaneously provide their data streams to composition of multiple 
overlapping images into individual full view frames unit 65. Unit 65 also receives conditional 
input from portable tracking control device 90 and provides its data stream to both center-of- 

10 play / choice of view analysis unit 66 and storage of tracking and video frames indexed by 

object and frame no. / time sequence unit 67. Choice of view analysis unit 66 in turn provides 
real time directives to automatic filming cameras 40 through conventional wire or optional RF 
modem 45m. Such directives are then input to computer directable event filming camera 45. 
Filming camera 45 then feeds its data stream back to storage and tracking unit 67 via 

15 conventional wire or optional RF modem. Unit 67 subsequently provides its data stream to 

quantification and analysis of object movement stored in cross-linked database unit 68. Unit 68 
optionally conducts two-way communications with portable tracking control device 90, team 
output stations 80 and remote computer system for reviewing captured events 70. The optional 
link with device 90 will be made via Rf connection (not depicted) while the link with remote 

20 system 70 will be made via remote communications devices 69 and 79. The information 

provided by local system 60 to remote system 70 will be provided to the end user via remote 
access to stored database of tracking and filming frames as well as object movement 
quantification and analysis unit 71. 

Operation 

25 Referring first to Fig.'s 1 and 5, normal operation of the preferred embodiment 

commences after the system 100 has been properly installed at an ice arena such as 2. One of 
the more critical aspects of installation is the attachment of the array of overhead x-y tracking 
camera assemblies 30c to the roof of arena 2. Mounting frame 50 has been designed to 
variably attach to rink girders 2d in such as way that the subsequently attached array of 

30 overhead assemblies 30c forms an overlapping field of view across of the ice surface 2 below. 
It is preferable that each assembly 30c maintains a perpendicular position with respect to 
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surface 2. It is anticipated that either fixed lenses with pre-selected depths of field or 
electronically controllable zoomable lenses will be used to properly establish the overlapping 
fields. The overlap is important to ensure that any object to be tracked can be followed from 
camera to camera as it moves throughout the predefined area. As a part of the initial 
installation and setup of system 100, a calibration procedure will be run to define the limits of 
each filming cameras' 25 and tracking cameras' 24 field of view. Once known, the system will 
restrict its searching to non-overlapping regions in order to ininimize duplicate processing. All 
assemblies 30c will either communicate with local system 60 via cables such as 26 or optional 
RF modems such as 25m and 24m. 

Referring now to Fig.'s 2a, 2b and 2c, the preferred embodiment provides for 
various methods of marking the objects to be tracked with a specially chosen frequency 
selective reflective material such as an ink. This ink is then used to embed into puck 3 as 
reflective ink 3a, to produce reflective tape 4a, to embed into markings of patches 7r and 71, 
and to produce reflective stickers 9 for helmets 8. It is also used to create cones 11 with 
reflective ink 11a as shown in Fig. 6. It is critical to the novelty of the present invention to 
note that this ink has been chosen specifically to reflect energy 23a as output by energy source 
23. This reflection has been depicted as 3b, 4b, 7m, 9a and lib and will be received by 
tracking cameras 24 through energy filter 24f . By using specially matched energy filter 24f on 
each camera 24, the amount of information required to be processed in order to follow all 
object motion is miiiimized thus greatly increasing the speed of deterrnining the object's 
position. 

To further illustrate the novelty of this point, Fig.'s 3a, 3b and 4a have been 
created to dramatize the resultant effects of the addition of this tracking ink. Fig. 3a shows a 
conventional camera system capturing ambient light and showing its corresponding field of 
view as scene 28 on monitor 27. Note that in this view there is considerably more information 
that must be processed including the object such as player 10 and the background such as ice 
surface 2. By adding reflective materials as previously discussed in the form of 3a, 4a, 7r, 71 
and 9, Fig. 3b now shows a much reduced amount of information in scene 28. Again, this is 
accomplished in real time without the requirements for additional computer processing by the 
application of energy filter 24f to tracking camera 24 which was specially chosen to pass a 
narrow band of frequency as emitted by energy source 23 as selected energy 23a. Energy 23a 
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radiates throughout the predefined area such as ice surface 2 and is reflected back as 3b, 4b, 
7m and 9a. Orientation marks 7rl and 711, team indicia 7r2, player indicia 712 as well as 
reflective sticker 9 have been produced to include special markings which will be easily 
discernable to indicia / object tracking analysis unit 62. The size of these markings will be 
5 matched to the target resolution of tracking cameras 24 in overhead assemblies 30c. Hence, the 
lower the resolution vs. field of view, the larger the marking will have to be to facilitate easy 
recognition. The preferred embodiment also includes unfiltered filming cameras 25 that will 
capture all ambient frequencies of light, as depicted in Fig. 4a. However, as will be discussed 
at a later point, due to information provided by the tracking analysis, only a small portion of 
10 this data must be gone through in order to extract all relevant object information. 

Prior to running system 100 it will also be necessary to install perspective z 
tracking camera sets 30. One set will be mounted at each end of ice surface 2 so as to be in 
constant view of the area immediately surrounding each goal 2b. These sets 30 are so placed in 
anticipation of the travel path of puck 3 which is expected to leave surface 2 and to travel in the 

15 Z direction when approaching goals 2b. Note that due to the overhead nature of assemblies 
30c, they will be unable to pick up any object movement in the Z dimensions which is 
acceptable and even desirable since it reduces the amount of processing that must be performed 
to track all objects. However, in the restricted area nearest to goals 2b, namely defined by 
triangular area 2c, it can be very beneficial to track the Z element of puck 3's path of travel. It 

20 is important to note that sets 30 and the information they collect are not required to allow the 
main aspect of what the present inventors feel is novel concerning this invention. Specifically, 
the practice of following movements of specially marked objects and to apply this information 
to normal video images to quickly extract object information. 

Also optional and yet novel to the present invention is the inclusion of 
25 automatic filming cameras 40 at discretionary locations within ice surface arena 2. Heretofore, 
camera operators who intuitively determined the center-of-play controlled the filming for 
broadcast of events such as hockey and self directed the pan, zoom and tilt of their filming 
cameras to capture the restricted scene. Now that system 100 will have all information 
concerning the movement of objects on the ice surface 2, it is planned that local computer 
30 system 60, via its center-of-play / choice of view analysis unit 66, will automatically direct the 
pan, zoom and tilt of the automatic filming cameras 40. 
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Once all overhead x-y tracking camera assemblies 30c, all perspective z 
tracking camera sets 30 and all automatic filming cameras 40 have been correctly installed and 
calibrated and all objects to be tracked have been appropriately augmented with marks such as 
3a, 4a, 7r, 71 and 9, system 100 is ready for operation. The ensuing discussion will describe 
how the movements of a single player 10, stick 4 and puck 3 will be tracked from frame to 
frame during the duration of the event across the entire predefined field of view in the X-Y and 
Z dimensions. Following this, the present inventors will teach how additional players 10 with 
their sticks 4 may also be simultaneously tracked and how problems due to object overlap, 
object change in size, object disappearance from view, sudden changes in ambient lighting and 
otherwise reductions in visibility for example because of fog will be handled. 

Referring now to Flg.'s 1, 2, 4a, 4b, and 8, the tracking of multiple objects by 
system 100 commences after the system has been turned on and the first player 10 proceeds 
down entrance-way 2e and crosses into the field of view of the array of overhead assemblies 
30c. Within entrance-way 2e player 10 will first appear in a single frame captured by video 
capture for tracking camera(s) unit 61. Unit 61 will in total capture 40 plus frames per second 
(40 + fps) from each tracking camera in assembly 20c and will feed each frame to indicia / 
object tracking analysis unit 62. 

Note that given the current state of technology one camera controller board 
such as would be required within unit 61 can control up to eight cameras. The actual number 
of frames per second that needs to be captured is dependent upon the desired movement 
resolution. For instance, the fastest object in our example is the puck 3 that will at times travel 
up to 100 mph. At 40 frames per second, 5280 feet per mile and 60 * 60 = 3,600 seconds per 
hour the maximum distance of travel for the puck 3 per second is: (100 miles/hour * 5280 
feet/mile) / (3,600 sec/hour) = 146.7 feet. At a rate of 40 frames per second, the maximum 
distance traveled between frames would be 3.7 feet. Note that a player such as 10 on skates 
will obtain maximum speeds of around 25 mph, or one quarter of the speed of the puck. At 40 
frames per second, the maximum distance traveled by a player 10 would be approximately 
0.925 feet. At 20 frames a second, the puck 3 will travel no more than 8 feet at maximum 
speed while the player 10 will be limited to approximately 2 feet in distance. Considering that 
both player 10 and puck 3 seldom travel at maximum speed, movement resolution will be 
considerably higher in practice. Further note that there is an anticipated area, namely around 
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goal 2b where puck 3 is most likely to reach top speeds. Both the overhead camera assemblies 
20c and the perspective camera sets 30 can be operated at an increased frame capture rate to 
increase movement resolution in these specific zones. In any case, the present invention has the 
ability to trade off cost of computing power versus movement resolution by trading off the 
5 frame rate and the number of tracking cameras 24 that will be attached to a single computer 
and video capture card. It should be recognized that the use of multiple computers to gather 
and analyze data is considered to be neither novel to, nor a limitation of, the present invention. 

As each frame from tracking camera 24 of overhead assembly 20cl is accepted 
by analysis unit 62, the grey scale of each pixel is compared against a threshold value where 

10 those pixels exceeding the threshold indicate the presence of any form of the special mark such 
as 3a, 4a, 7r, 71 and 9. Since every player 10 must both enter and leave the ice surface 2 from 
entrance- way 2e which is always in view by assembly 20cl, and since the maximum 
movement between frames can be calculated based upon the anticipated maximum speed and 
the frame capture rate, it is possible to calculate a minimum number of pixels that must be 

15 completely searched in order to first detect the presence of a new player such as 10. This 
minimum number of pixels is shown as lOe in Fig. 4c and consists of those pixels running 
parallel along the outermost edge of assembly 20cl's field of view inward several rows toward 
the main ice surface 2. The depth of this rectangular area will again depend upon the maximum 
distance a player 10 can be expected to travel between frame captures. Note that even in other 

20 sporting events such as basketball or football, there is usually a tunnel through which each 
team must travel to gain access to the main arena. If the systems field of view is properly 
extended to be continuous into any such tunnel, then this minimum search technique may be 
employed to first detect any new player to be tracked. As will be shown, once detected, the 
processing requirements of continuing to track are significandy reduced since the extent of a 

25 player's movement between frames is limited. 

Referring now additionally to Fig. 4c, once a marked object is detected, 
whether it be a patch 7r or 71 on pads 6, or the reflective sticker 9 on helmet 8, or the 
reflective ink 3a on puck 3, or the reflective hockey tape 4a wrapped around a stick blade 4, it 
is tracked individually. The first detected movement of each mark is expressed in polar 
30 coordinates relative to the center point 2a of ice surface 2. Hence each object's first movement 
is expressed as an angle off 0° and a distance from center point 2a along the stated angle, these 
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calculations are performed constantly by object tracking analysis unit 62. Once unit 62 has 
detected a given object, it will continue to look for that object within its field of view based 
upon the last known coordinates, the last know vectors of direction, acceleration and velocity 
and the calculated maximum distance of travel between frames. The last known coordinates 
combined with the maximum possible travel distance will work to define a maximum bounding 
circle, depicted as 10b in Fig. 4c, of possible movement which must be searched in order to 
find the objects next location. In order to cut the average search time within this bounding 
circle, unit 62 will first look in the last known direction of travel based upon prior movements 
out a distance equal to the last know rate of speed divided by the known frame rate. From this 
new most likely point within the bounding circle, unit 62 will continue to search by growing 
this point in all directions until the entire bounding circle has been analyzed. In the event that 
the marking is not found and it is known that the object has entirely passed through the 
minimum rectangle of pixels bordering the edge of the field-of-view covering the entrance-way 
2e, then the object will be searched for on the next frame. In this case the new maximum 
bounding circle's radius will be twice that of the previous search. 

If a marked object is detected and then is lost to system 100, tracking analysis 
unit 62 will first communicate with object / scene extraction analysis unit 64 to determine if the 
presence of additional ambient light based information will uncover the objects location. Before 
reviewing this technique, it should first be noted that once a marked object is detected, its type 
and location are passed to extraction unit 64 from tracking unit 63. extraction unit 64 then 
analyses the corresponding unfiltered video frame taken by the filming camera 25 that is 
housed in the same overhead assembly 20c whose tracking camera 24 is currently viewing the 
located object. Knowing the type of object, for example a patch 7r or reflective sticker 9, 
indicates to extraction unit 64 the maximum expected size of the actual object, for example 
pads 6 or helmet 8. Given this maximum size along with the current location of the reflective 
indicia, unit 64 will begin at the location of the attached indicia and then work its way out 
away from the indicia comparing each pixel of the image to a pre-known background 2r, as 
depicted in Fig. 4b. Once it is determined that background pixels have been reached, the 
image-tracing path lOp continues around the object until a closed path is completed. Around 
this closed path lOp a minimum bounding rectangle lOr is calculated in order to quickly 
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extract the portion of the video image containing the tracked object. By knowing the expected 
maximum size of the object associated with the type of indicia found object (e.g. pad 6 or 
helmet 8 or player 10), this process can be restricted to a maximum bounding circle in order to 
find the edge of the object. Note that this object defined by closed path lOp may and often will 
5 contain two or more indicia such as patches 7r and 71 as well as sticker 9. This extracted 

surface as defined by lOr, as well as the tracked location of each reflective object within that 
surface is then passed to composition of multiple overlapping images into individual full view 
frames unit 65. Unit 65 then catalogues into one complete set all detected reflective objects 
such as 3b, 4b, 7m and 9a and their corresponding extracted object pixels that have been 

10 detected over the entire field of view. As a matter of practice, the extracted pixels will be 
expressed as the minimum-bounding rectangle as depicted by lOr rather than a list of pixel 
coordinates. Within this minimum bounding rectangle, all background pixels will have been set 
to a null value by extraction unit 64 to clearly differentiate them from the foreground object 
during later examination. A center of gravity as well as polar coordinates to that center point 

15 are also calculated by Extraction Unit 64 and passed to composition unit 65. And finally unit 
64 will determine a beginning edge point to be associated with each object that can be used by 
later routines to quickly trace the outline of the object from within its minimum bounding 
rectangle to perform a final extraction. It is also the responsibility of composition unit 65 to 
piece together reflective surfaces such as 3b, 4b, 7m and 9a as well as the objects to which 

20 they are attached such as pads 6 or helmet 8 or player 10 which may overlap frames taken by 
separate filming 25 or tracking cameras 24. 

Once the composition unit 65 has created the known set of polar coordinates to 
the centers of gravity for all know reflective markings and their corresponding objects and has 
also defined the minimum bounding rectangles and a beginning edge point this entire set of 

25 data is passed to the storage of tracking and video frames indexed by object and frame no. / 

time sequence unit 67. As the entire set of frames across the field of view created by the array 
of overhead assemblies 20c and perspective sets 30 are continually input by capture units 61 
they will be passed to tracking unit 62 which will follow each new object as it enters the field 
of view 20cl and ultimately leaves via the same assembly 20cl. The present inventors 

30 anticipate that after identifying the first appearance of an inked surface or corresponding object 
it may be more efficient to allow tracking unit 62 to express subsequent centers of gravity 
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using the same polar coordinate method except that the center point is no longer the center of 
Ice 2a but rather the previous center of gravity for the same inked surface or corresponding 
object. This changing center of reference strategy is shown in Fig. 4c by the sequence lOal 
through 10a4. 

Storage unit 67 will take in this continual flow of inked surface and 
corresponding object information and will create several databases for later analysis. First, 
each individual surface and object will have its own movements catalogued from the point it 
enters the field of view until at some later point it leaves. Note that the field of view is 
extended to cover team boxes 2f and 2g as well as the penalty box 2h. Hence, to system 100, 
there is no difference to tracking these surfaces and objects in these areas, where they are not 
active in the current event, as opposed to on the ice surface 2, where they are active, storage 
unit 67 is also responsible for creating and storing group information. For instance, unit 67 is 
expecting to find the following associations: 

1. It will always true that a player 10 will have an associated right patch 7r 
and left patch 71. 

2. It will most often be true that a player 10 will also have an associated 
helmet 8 with sticker 9 and a stick 4 with tape 4a. 

3. It will sometimes be true that a player 10 will have puck 3 within its 
direct semi-circle of control. This semi-circle will be defined by the player 10's current 
location, direction, speed of travel and orientation and is depicted as 10s in Fig. 4c. 

4. It will sometimes be true that a player 10 may appear to have additional 
patches, helmets or sticks in which case multiple players may have collided. 

Whatever associations are detected are used to form a players group database. 
Note that storage unit 67 will distinguish between a stray stick or helmet and a player. As such, 
the stray stick or helmet will not create another instance of a player group. Hence, frame by 
frame a count will be made of to total number of unique player groups which should stay equal 
unless: 

1. A new player enters the field of view. 

2. An existing player exists the field of view. 
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3. There is a collision between one or more players such that their forms 
are temporarily joined. 

When two or more player groups are joined to form a single group somewhere 
within the field of view, it is anticipated that eventually the players will separate. At this point 
5 the system which had assigned the groups movements to each know player who had entered 
the group, will now begin to track individual players and the total group count will have 
returned to its pre-collision count. 

As storage unit 67 continually updates its various databases, this same 
information then becomes available in real time to quantification and analysis of object 
10 movement stored in cross-linked database unit 68. Quantification and analysis unit 68 will have 
a variable set of tasks it can perform on the tracking databases which can be enabled as either 
real-time or post event. In either case, some of those tasks are listed as follows: 

1 . Correlate individual player group data to a pre-known list of potential 
players. This is accomplished by comparing the team indicia 7r2 as well as the player indicia 

15 712 to a pre-known database of values. 

2. Determine player orientation based upon the location of inked surfaces 
within the extracted objects in combination with the last known direction, speed and orientation 
of both the inked surfaces and their associated objects. 

Hence unit 68 will be able to identify that a player is skating backward versus forwards, or that 
their head and shoulders are beginning to turn and therefore it is expected that so will their 
direction of travel in the next captured frame. It should be further noted that reflective sticker 9 
was specifically designed to also assist unit 68 in determining whether the associated helmet 8 
(and therefore players head) is looking down or straight ahead. This is facilitated by the 
alternating black and white squares of sticker 9 that will vary in count depending upon helmet 
8's orientation. 

3. Update player statistics such as time entered and left ice surface (known 
as a shift), duration of shift, average speed, percentage of shift spent in defensive, neutral or 
offensive zones, percentage of puck control, number of shots, turnovers, steels, passes, etc. 
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4. Update team statistics such as number of shifts, average duration of 
shifts, percentage of play in defensive, neutral and offensive zones, percentage of puck control, 
number of shots, turnovers, steels, passes, etc. 

Any and all of this information that was pre-enabled to be tracked in real time 
is now available for inquiry and display on team output stations 80. After the event has 
completed, or during scheduled breaks in event activity, quantification and analysis unit 68 
may continue to process the stored databases in order to derive additional information which 
was not pre-enabled for real time. Other events, whether they are sporting or not and 
irrespective of the inclusion of human objects, will have their own set of uniquely desirable 
quantification and analysis requirements. 

At any time after the completion of the event and all of its analysis by unit 68, 
it may be desirable for selected individuals to remotely inquire upon the information tracked 
and analyzed by local computer system 60. Via the use of conventional remote 
communications devices depicted as 69 and 79 in fig. 8, a remote computer system for 
reviewing captured events depicted as 70 may be used to access the event databases. The 
present inventors do consider that the disclosed technique of first tracking specially inked 
surfaces which have been attached to objects which are desired to track and then extracting the 
minimum bounding areas of those objects movements is novel for at least the following 
reasons: 

1 . The technique operates in a minimum of time and requires a minimum 
of computer processing by using a selected and narrow band of energy to illuminate and 
capture a small amount of information which yields object identity, orientation, direction of 
travel and speed at a minimum, irrespective of the ambient background reflection. 

2. The technique employs the quickly determined information from the 
narrow band of energy to efficiently extract a larger set of information from the entire band of 
energy typically found in ambient light. 

3. The technique creates a resultant movement database which can be used 
to reconstruct one or more portions or all of the event after it has concluded by overlaying all 
captured and extracted object movements back onto a pre-known ice surface background. 
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This last point is especially beneficial since in practice it allows what otherwise 
would be a prohibitive amount of event video information to be saved as a minimum set of 
movement information which can then be transferred via normal system to system connections 
such as the internet. Thus several individuals involved with an event can separately and easily 
5 download this data to their respective remote systems 70 for there own selective replay and 

analysis. It is even envisioned that only the centers of gravity need to be transmitted as opposed 
to all object pixels in the minimum bounding box since in a arcade style event replay the 
graphics of a players body, helmet and stick as well as the puck can easily be generated by the 
local system. 

10 An additional novel aspect of system 100 is depicted in Fig. 6. Specifically, by 

the use of pre-designated marked objects such as cone 11 with reflective ink 11a, it is possible 
to restrict system 100's field of tracking to some sub-portion of its field of view such as 2t. In 
practice an operator such as an on-ice coach will initiate this reduced tracking mode of system 
100 using a special entry code input to tracking control device 90. Device 90 transmits this 

15 control sequence via conventional RF to local computer system 60. This information is then 
provided to composition of multiple overlapping unit 65 that then searches for two or more 
pre-designated objects, e.g. cones 11, on the ice surface 2. When taken together, these two or 
more cones 11 will prescribe one or more geometric areas such as 2t within which all tracking 
is enabled and outside of which all tracking is disabled. In those cases where the cones have 

20 been placed in an ambiguous fashion, unit 65 will transmit for display all possible layouts 

given the current cones 11 detected from which the operator of control device 90 will select the 
appropriate layout. For example, the coach may simply place two cones 11 on Ice 2, one at 
either side of the red line that divides the ice surface into two parts. At this point unit 65 will 
display two possible layouts covering either one side of the ice or the other. 

25 After restricting system 100's tracking area, all other functionality is identical 

as previously described. Furthermore, quantification and analysis unit 68 is capable of 
outputting all calculated information to portable tracking control device 90 in response to the 
on-ice coaches real time inquiries. Unit 68 can be enabled by device 90 to also output the speed 
of successive shots on goal to a shot clock 91. 
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Thus the reader will see that the Multiple Object Tracking System provides a 
novel apparatus and method for: 

1- tracking multiple, fast moving, variably shaped objects such as humans 
in real time without the aid of operator intervention to either initially identify or conditionally 
re-identify the objects during the tracking period; 

2- reducing the time required to process image data by first separately 
tracking a special indicia which has been attached to the object after which the indicia's 
location may then be used along with the vectors of direction, acceleration and velocity to 
efficiently extract the object from its background; 

3- performing its recognition tasks independent of color distinctions within 

the objects; 

4- performing its recognition tasks independent of either passive or active 
electronic tracking devices; 

5- determining orientation information about the objects as a whole and 
potentially of their individual parts; 

6- performing under various lighting and air moisture conditions which 
may not be optimal for recognition techniques based upon visible light; 

7- easily constricting the field of tracking within the field of view; 

8- creating and maintaining in real time a movement database accessible 
by either time and or object(s) identity; 

9- outputting either all or some of the captured data either immediately 
within the ice arena of after the event to one or more remote computers over a communication 
network such as the Internet; and 

10- providing information to remote computer systems over the Internet that 
may be used for controlled event replay as well as critical analysis. 

While the above description contains many specifications, these should not be 
construed as limitations on the scope of the invention, but rather as an exemplification of 
preferred embodiments thereof. Many aspects of the systems functionality are beneficial by 
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themselves without other aspects being present. For instance, the separate filming cameras 
could be dropped as a matter of cost savings and / or speed improvement. The system would 
still then provide significant and novel benefit by tracking the inked surfaces attached to the 
objects to be tracked. There does not have to be an array of tracking cameras that are mounted 

5 directly above the event be tracked. If this arrangement proves to inconvenient, then the 

cameras simply need to be placed so that together they can create an overlapping field of view 
that is most likely to keep the ink surfaces in view. For instance, on the set of a movie 
production, this system could be used to automatically follow one or more aspects of the scene 
as the action unfolds. Hence, if the actors were to wear large marking made of the special ink 

10 on patches hidden underneath their clothes, than this tracking system could follow the actors 
movements and automatically direct the pan, tilt and zoom of selected production cameras. 

Another example of an aspect of the system that is beneficial but not mandatory 
is the link to a remote computer system for reviewing the captured events. While the present 
inventors do consider this system to be unique in the way it stores information which is 
15 especially beneficial for remote download, the remote system is not necessary for the system to 
have novelty and usefulness. 

It is evident from the description of the multiple object tracking system that it 
has applicability beyond that of tracking the movements of hockey players and the puck during 
an ice hockey game. For example, this same system could be set up over an outside roller 

20 hockey rink if the frame which holds the overhead assemblies where it self to be mounted on 
posts or polls to hold it above the playing area. The system could also be used to track 
basketball in a fashion very similar to ice hockey since these games are nearly always played in 
an indoor arena. Similar approaches could be used with other sports such as football and 
baseball as long as the field of view is sufficiently covered with perspective tracking cameras 

25 because there will not be any overhead assemblies. The system could also be used in large 
convention halls or auditoriums to track security whereabouts, attendee flow of support staff. 
This could be accomplished by using the same overhead tracking cameras while the filming 
cameras would more than likely be unnecessary. Each type of person to be tracked could be 
asked to where a special patch that could even be coded based upon statistically relevant 

30 criteria as determined by the event hosts. As individuals with patches moved about and visited 
different booths, their choices could automatically be tracked including their time spent at each 
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selected both. Such a system could also help with crowd flow if large lines where detected as 
forming around selected areas. Note that in this application, it is less critical that each any 
every movement of each and every person to be tracked is followed, but rather that in total the 
majority of movements of all like individuals are determined from which helpful decisions and 
statistics might be derived. 

From the foregoing detailed description of the present invention, the Multiple 
Object Tracking System, it will be apparent that the invention has a number of advantages, 
some of which have been described above and others of which are inherent in the invention. 
Also, it will be apparent that modifications can be made to the Multiple Object Tracking 
System without departing from the teachings of the invention. Accordingly, the scope of the 
invention is only to be limited as necessitated by the accompanying claims. 
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In tlie Claims 

We claim: 

1 . An automated system for tracking the movement of multiple objects 
within a predefined area comprising: 

5 means for attaching special markings onto said objects where 

said markings are selected to be reflective to a selected narrow band of 
energy, 

means for radiating said narrow band of energy throughout 
said predefined area, 

10 means for receiving reflections of said radiated energy off of 

said special markings, and 

means for determining said location of said special markings 
and therefore said objects based upon said reflections. 

2. An automated system for tracking the movement of multiple objects 
15 within a predefined area comprising: 

means for attaching special markings onto said objects, 

means for tracking the location of said special markings, and 

means for employing said location of said special markings 
to extract the image of said objects from said object's background. 

20 3. An automated system for tracking the movement of multiple objects 

within a predefined area comprising: 

means for establishing a field of view which encompasses 
said predefined area, 

means for first minimally searching a portion of said field of 
25 view where said objects are expected to enter and exit said area, 

means for establishing said objects first position within said 
field of view based upon said first minimally searching, and 
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means for subsequently minimizing the field of view which 
must be searched to find said objects next location based upon said objects 
first and subsequently prior know position, it last calculated direction 
vector and its pre-known maximum rate of travel. 

4. An automated system for remotely providing selective information 
regarding the movements of multiple objects within a predefined area comprising: 

means for establishing a field of view which encompasses 
said predefined area, 

means for marking each of said objects with a unique 

10 indicia, 

means for tracking said indicia, 

means for establishing a minimum database of the 
movements of said unique indicia cross-linked to said marked objects, and 

means for selectively transmitting some or all of said cross- 
linked database to a remote system for subsequent analysis. 

5- An automated system for tracking the movements including direction 
and orientation of multiple objects within a predefined area comprising: 

means for establishing a field of view which encompasses 
said predefined area, 

means for marking each of said objects with a unique 
indicia, 

means for tracking said direction and said orientation of said 
unique indicia, and 

means for determining said direction and said orientation of 
25 said objects based upon said direction and said orientation of said unique 
indicia. 

6- A method for efficiently extracting an object from its background image 
as it changes its position from frame to frame comprising the steps of: 



15 



20 
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marking said object with a special frequency selective 
reflective material, 

radiating upon said object an energy whose frequency 
matches the reflective properties of said material, 

receiving reflections of said radiated energy off of said 
material through a camera fitted with a filter designed to pass the 
frequencies matching said radiated energy, 

determining the location of said reflective material by 
searching images captured by said filtered camera, 

capturing time and location matching images of said object 
with an unfiltered camera, and 

efficiently extracting said object from said matching images 
based upon said determined location of said reflective material. 
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(57) Abstract: A system (100) for tracking the 
movement of multiple objects within a predefined 
area using a combination of overhead X-Y filming 
cameras (25) and tracking cameras (24) with 
attached frequency selective filter (24f). Also 
employed are perspective Z filming cameras (25) 
and tracking cameras (24) with filter (24f). Objects 
to be tracked have been marked with a frequency 
selective reflective material, such as patches (7r and 
71 ), sticker (9) and tape (4a). System (100) radiates 
selected energy (23a) throughout the area of tracking 
to reflect off the reflective materials. Reflected 
energy such as (7m, 9a and 4b) is then received 
by tracking cameras (24) while all other ambient 
light is blocked by filter (24f). Local Computer 
System (60) captures images from tracking cameras 
(24) and locates said markings. Using the location 
information along with preknowledge concerning 
the multiple objects maximum rate of speed and 
maximum size as well as calculated movement 
information, system (60) is able to extract from the 
background the portion of the unfiltered images that 
represent the multiple objects. 
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